Cut database and hosting costs: archive snapshots to R2, fix the write path - #16
Merged
Conversation
GameServerSnapshot_createdAt_idx (4.8 GB) is redundant with the (gameServerId, createdAt) composite; PlayerInfoMap_playTime_idx (805 MB) and ClanInfoMap_playTime_idx (148 MB) are unused. On production these are dropped manually with CONCURRENTLY and the migration marked applied — see docs/cost-reduction-runbook.md, which also carries the rest of the manual ops steps for the plan. Co-Authored-By: Claude Fable 5 <[email protected]>
The nested create on the @unique relation made Prisma disconnect the old state row (SET gameServerId = NULL), insert a new one, and cascade-delete the orphan's clients — ~3 row versions per state row and 2N per client row, 255k times a day, on tables holding ~900 live rows, churning the GIN trigram index on name. The in-place upsert lets Postgres HOT-update the state row, and the global orphan sweep (a lock-contention hotspot across up to 300 concurrent pollers) is no longer needed since orphans can no longer be created. The targeted delete in the failure branch stays. Co-Authored-By: Claude Fable 5 <[email protected]>
Orphaned states can no longer be created now that pollGameServer upserts the state in place. Deletes any existing orphans and adds NOT NULL. Note: CI applies migrations before deploying the worker, so during the deploy of this change old workers briefly fail polls on the NOT NULL constraint. Harmless and self-healing once the worker image rolls. Co-Authored-By: Claude Fable 5 <[email protected]>
New archive-snapshots worker (10-min tick, concurrency 1): resolves a watermark id older than SNAPSHOT_RETENTION_HOURS (default 48h), then batches snapshots + clients into flat Parquet files (zstd, one row per client-observation, sorted by gameServerId/createdAt, Hive-partitioned by day) uploaded to S3-compatible storage — MinIO locally, R2 in prod. Rows are deleted only after a successful HeadObject, clients explicitly before snapshots to avoid per-row cascades. A wall-clock budget (ARCHIVE_TIME_BUDGET_MS) lets the same code path drain the 180M-row backlog incrementally and then idle in steady state. Writer is parquet-wasm (no native module — the worker image is musl) with apache-arrow for table construction; round-trip integrity (int64 precision, ms timestamps, nulls) is covered by parquet.test.ts. docker-compose gains MinIO + bucket init so the whole path runs locally. Co-Authored-By: Claude Fable 5 <[email protected]>
The playerInfoGameType / clanInfoGameType / playerInfoMap / clanInfoMap
loops were pure existence checks (update: {}). Every one of those rows is
also created by updatePlayTime and the rank methods, which are enqueued
for every successful poll — and gameServerScheduler skips polling
entirely when those queues are full, so the jobs can't be silently
dropped. This removes 4 × N statements per poll.
Kept as its own commit: this is the item most worth reverting in
isolation.
Co-Authored-By: Claude Fable 5 <[email protected]>
N round trips per poll become one INSERT ... ON CONFLICT DO UPDATE. Rows are sorted by name (unordered multi-row upserts deadlock under concurrent pollers with overlapping player sets — same for the clan createMany above it), createdAt/updatedAt are set explicitly since they're Prisma-managed, and the DO UPDATE is a no-op unless lastSeenAt moved by more than 10 minutes or the clan changed, roughly halving Player updates that each cost up to 4 index writes. Co-Authored-By: Claude Fable 5 <[email protected]>
updatePlayTime went from ~50 round trips per snapshot (seven sequential
per-client upsert loops plus three updates) to at most seven multi-row
statements, each sorted by its conflict key to avoid deadlocks between
concurrent workers. It also early-returns when deltaPlayTime is 0, which
previously wrote zero-increment rows to all seven tables.
The rank methods now read existing info rows with one findMany (creating
the rarely-missing ones first) instead of upserting per client — an
update:{} upsert still writes a dead heap tuple per row — and apply
rating changes as one multi-row UPDATE ... FROM (VALUES ...) per table.
PlayerInfoMap is the main beneficiary: 12 GB with 4.2M dead tuples (9%),
and the largest table once snapshots are archived.
Co-Authored-By: Claude Fable 5 <[email protected]>
Every statement was previously its own implicit transaction — its own commit and WAL flush, ~45 fsyncs per poll. Array form deliberately: the interactive callback form pins a connection for its whole duration. map.upsert stays outside because its id is needed to build the other statements. Also stops reading the created snapshot's clients back — only the id was used. Co-Authored-By: Claude Fable 5 <[email protected]>
…gint The /status page (force-dynamic) counts servers by master server and was seq-scanning all 18k GameServer rows on every hit — Prisma doesn't auto-index relation scalars. GameServerStateClient.id burns ~400k sequence values a day from the state client delete/recreate cycle and would eventually overflow int4; the table holds under a thousand rows so the type change is sub-second. Co-Authored-By: Claude Fable 5 <[email protected]>
Prisma binds integer template parameters as bigint, and make_interval(hours => bigint) doesn't exist. Caught by the local end-to-end archive run. Co-Authored-By: Claude Fable 5 <[email protected]>
Review feedback: drop the as PlayerInfoMap[] casts by constructing fully typed rows, and remove a redundant comment. Co-Authored-By: Claude Fable 5 <[email protected]>
Co-Authored-By: Claude Fable 5 <[email protected]>
Review feedback: the pure Prisma TS API has no multi-row upsert (per-row upsert() is exactly the round-trip pattern being removed), but typedSql — already used for search — keeps one statement per table with generated, type-checked signatures. Each batch becomes a .sql file under libs/prisma/prisma/sql taking parallel unnest() arrays, called via prisma.$queryRawTyped. The elo increments now COALESCE a NULL rating to 0: previously whichever of updatePlayTime/rankPlayer created the info row first decided whether ratings could ever accrue (NULL + delta stays NULL). Verified against a live database: player upsert (including empty-clan handling), all seven playtime statements, and both rank methods. Co-Authored-By: Claude Fable 5 <[email protected]>
Co-Authored-By: Claude Fable 5 <[email protected]>
Verified empirically: PostgreSQL rejects DROP INDEX CONCURRENTLY inside a transaction block (SQLSTATE 25001), and prisma migrate deploy (5.22) wraps multi-statement migrations like this one in a transaction — a scratch migration with two concurrent drops fails and rolls back, while a single-statement one is sent unwrapped and succeeds. Co-Authored-By: Claude Fable 5 <[email protected]>
The combined migration would have run plain DROP INDEX on prod under an ACCESS EXCLUSIVE lock. Prisma (verified on 5.22) sends single-statement migrations unwrapped, so splitting into three migrations of exactly one DROP INDEX CONCURRENTLY each lets migrate deploy do the drops lock-free on merge — no manual psql, no migrate resolve step. Verified by replaying all 66 migrations on a fresh database: the three drops apply, the indexes are gone, and migrate diff reports no drift. Co-Authored-By: Claude Fable 5 <[email protected]>
- fillfactor + eager-autovacuum tuning becomes a normal transactional migration (storage parameters are catalog-only and invisible to Prisma drift detection — verified with migrate diff). - CI pins the worker fleet to one machine after each deploy. - The archive worker idles with a log line in production until S3_ENDPOINT is set, so deploying before the R2 bucket exists is safe; provisioning R2 credentials via fly secrets is the single remaining out-of-repo action. - Deliberately dropped from the plan, given weeks (not days) of headroom: the manual GameServerClient_pkey drop (space returns at the Phase 3 dump/restore anyway) and VACUUM FULL/REINDEX on the <5 MB state tables (autovacuum plus HOT updates cover it). The runbook is now a description of what merging does, not a checklist of things to type. Co-Authored-By: Claude Fable 5 <[email protected]>
Co-Authored-By: Claude Fable 5 <[email protected]>
lodash is already a dependency (uniqBy in the same worker); sortBy's comparator is the same deterministic code-unit ordering, which is all the deadlock-avoidance sort needs. Removes three copies of the helper plus an inline comparator, and the multi-key clanPlayers sort becomes sortBy(..., ['clanName', 'playerName']). Co-Authored-By: Claude Fable 5 <[email protected]>
R2 is provisioned and the worker's secrets are set, so the archive job no longer idles when S3_ENDPOINT is missing — a misconfiguration now fails the job visibly in the queue. Co-Authored-By: Claude Fable 5 <[email protected]>
Co-Authored-By: Claude Fable 5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Frees the production database from its 90%-disk read-only cliff and removes most of its write load, all code-driven. Three lock-free single-statement `DROP INDEX CONCURRENTLY` migrations reclaim ~5.8 GB on merge, and a new archive worker continuously drains the two snapshot tables (69 GB, 79% of the database) to Cloudflare R2 as zstd Parquet — flat rows, Hive-partitioned by day, deleted only after verified upload — with credentials already set on the worker. The poll write path is rewritten: GameServerState is upserted in place instead of disconnect-and-recreate, redundant ensure-exists loops are gone, per-row upserts across Player/playtime/rating tables become single multi-row typedSql statements sorted by conflict key, and each poll commits in one transaction (~45 fsyncs → 1). Schema fixes add the missing `masterServerId` index and widen `GameServerStateClient.id` to bigint before it overflows. Verified end to end: full test suite, Parquet round-trip tests, and a live local run (1,271 real servers, ~1,900 polls, zero failed jobs) with the archive readback matching row-for-row — see `docs/cost-reduction-runbook.md` for what merging does.
🤖 Generated with Claude Code