Skip to content

M16.1 Record and count API requests #64

Closed

Description

@productdevbook

Part of #63. Plan in docs/architecture.md › API monitoring (M16).

yuva-api

  • Contract: kind (site, api) on analytics properties with apdex_ms and monitor_hour_days; POST /v1/analytics/properties/{id}/requests (1–1,000 records, analytics:write, gzip, 202 with accepted and errors); dataset: api in the query operation with the M16.1 metrics (requests, requests_per_minute, errors, error_rate, client_errors, client_error_rate, p50, p95, p99, avg_duration) by route, method, status, status class, environment and time:minute|hour|day, and ranges 15m, 1h, 6h, 24h. Lint, regenerate.
  • Records: method, route or path, status (0 for no response), duration, request and response size, time, optional consumer, API version, environment, service, release, ip, sample_rate. No body, header, query string or user agent accepted; the address shortened, looked up for the country and dropped in the first function.
  • Route inference from path (ids, UUIDs, dates, long hex or base64url, e-mail-shaped segments → {id}), 404/405 without a pattern → (unmatched), route rules, caps with (other).
  • Duration sketches (base 2^(1/32), sparse varint bytea) and size sketches (scale 2) with merge, quantile and Apdex in Go, with tests against exact percentiles.
  • Per-process aggregation with a random writer id, cumulative upserts every 10 s, the last flush 2 minutes after a bucket ends, late records under a fresh writer, drain on shutdown.
  • Tables with workspace_id: monitor_minute (day partitions, 8 days), monitor_hour, monitor_dim_hour (month partitions), monitor_day, monitor_dim_day, monitor_routes, monitor_names, monitor_rollup_hours, monitor_ingest_stats; partitions in the aggregator and analytics_partitions; the monitor_rollup fold on the analytics queue.
  • usage_counters.monitor_requests, YUVA_MONITOR_MONTHLY_REQUESTS, yuva workspace monitor-limit, YUVA_MONITOR_RECORDS_PER_MINUTE, YUVA_MONITOR_BUFFER_KEYS, YUVA_MONITOR_MINUTE_DAYS, YUVA_MONITOR=off.
  • sdk/go/analytics: Middleware for net/http that reads r.Pattern after the mux ran (method and host stripped) or uses a Route function, batching 500 records or 5 seconds through the existing queue and backoff, optional SampleRate for successful requests only.
  • Load test: 300M records a month on one property (500 routes, 5,000 consumers, bursts of 2,000 records a second): ingest p99 under 100 ms, data visible within 15 s, 30-day overview p95 under 1 s.

yuva-panel

  • Creating an API property; API overview: requests, error rates and p95 over time, the last 60 minutes by minute; routes table sorted by traffic, errors or latency; a route's page with its history and status codes; caps reached shown. en and tr.

yuva-docs

  • "API monitoring" guide with the Go middleware and curl, what is and is not recorded; en, tr, de.

Done when

A Go service with the middleware shows its routes in the panel within a minute, with counts and error rates equal to the requests sent and p50, p95 and p99 within 1.1% of the exact values, and the next day the same numbers from folded days; a raw path with an id never appears as its own route and 404 probes stay in (unmatched); Postgres holds no row per request and no IP address; a key without analytics:write gets 403; the load test meets its targets.

Activity

  1. productdevbook commented on Oct 14, 2026

    @productdevbook
    OwnerAuthor

    API part shipped locally (634d9d2, 4fcac89, a2f178e, afc19d6); load test at 2,000 records/s against 30 days of history: ingest p99 31.7 ms, visible after 10.1 s, 30-day overview p95 515 ms.

    Still open in this step:

    • An endpoint to hide a route (the hidden column exists).
    • A notice when the monthly request limit is reached (records past it are only counted as over_limit).
    • A time zone change applies to hours folded after it, not from the next local day as docs/architecture.md says.
    • Panel and docs parts.
  2. productdevbook commented on Oct 14, 2026

    @productdevbook
    OwnerAuthor

    This step's code is no longer on main; it lives in #72.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions