Skip to content

docs: the container never slept - the ~8.5min tail was a backfill artifact (#120) - #155

Merged
wormeyman merged 1 commit into
mainfrom
container-never-sleeps-measurement
Aug 5, 2026
Merged

docs: the container never slept - the ~8.5min tail was a backfill artifact (#120)#155
wormeyman merged 1 commit into
mainfrom
container-never-sleeps-measurement

Conversation

@wormeyman

Copy link
Copy Markdown
Owner

The number in #120 was an artifact of reading the dataset too early

#120 published an ~8.5 minute idle tail, measured from containersUsageAdaptiveGroups about 14 minutes after the render that produced it. 22:50Z was recorded as absent - asleep.

Re-read now that the dataset has settled, that bucket is there, and so is every bucket after it. The placement that render woke - a20877a2, first bucket 2026-08-03T22:35:00Z - was still allocated 29.3 hours later:

five-minute buckets present 352 of 352
gaps > 5 min none
distinct placements 1 (a20877a2)
total allocated memory 103,575 GiB-seconds
awake fraction (1 GiB instance) 0.981
worker requests in that window 5

Five requests cannot hold a container open for 29 hours against sleepAfter = "20s". There is no tail. The container never sleeps at all - which is precisely what the SIGTERM diagnosis in #154 predicts: @cloudflare/containers sends SIGTERM and never escalates to SIGKILL, node is PID 1 and ignored it, so the instance only ever goes away when a deploy replaces the placement.

Two cost claims in CLAUDE.md were wrong

1. The drain fix did not end the bill. Daily allocated memory, applicationId = a039bbb1, against the 96.0 GiB-hours a 4 GiB instance bills for a full day:

2026-07-13  95.6    2026-07-25  95.6    2026-07-31  95.7
2026-07-14  96.2    2026-07-26  97.0    2026-08-01  96.3
2026-07-15  98.2    2026-07-27  95.3    2026-08-02  99.0
2026-07-20  96.3    2026-07-28  96.1    2026-08-03  89.1 (downsize at 21:22Z)
2026-07-21  96.0    2026-07-29  87.0    2026-08-04  23.6 (= 24.0 at 1 GiB)
2026-07-22  96.8

100% of every full day, including the five days after the drain fix deployed on 2026-07-29. The ~$28/month was still being paid right up to the 2026-08-03 downsize, which cut it ~4x rather than ending it. The drain guard stays - the hazard is real and worker.spec.ts should keep testing it - but it was never the load-bearing cause of the bill.

2. The disk-to-memory ratios are dashboard units. Queried in bytes, which is what the GraphQL dataset actually returns, standard-1 reads 1.86 and basic reads 3.73, not 2.0 and 4.0. Both forms are now stated so neither reading surprises anyone.

What this PR does

Docs only - no behaviour change. It corrects those two claims and adds two bullets that were missing: the SIGTERM/PID-1 cause, and the backfill trap ("wait at least an hour before reading absence as sleep, and confirm with placementId continuity rather than bucket presence alone").

Status of the fix itself

Still not deployed. 343df53 is on main; production runs worker version 2e67470f and container app version 3 (image sha256:16205eda..., updated_at 2026-08-03T21:24:01Z), both of which predate the handler. #120 stays open until a preview:deploy and a settled re-read of the buckets.

Verification

pnpm run verify - exit 0. 1588 app tests passed / 3 skipped (196 files), 13 worker tests, 4 container tests including exits promptly on SIGTERM, 0 warnings.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QZvNS2H4cbaZk46ybA7hbj

…ifact (#120)

#120 published an "~8.5 minute" idle tail from `containersUsageAdaptiveGroups`,
read ~14 minutes after the test render that produced it. Re-read now that the
dataset has settled, the same window tells a different story: the placement that
render woke (`a20877a2`, 2026-08-03T22:35Z) was still allocated 29.3 hours later
- 352 of 352 five-minute buckets present, zero gaps, 98.1% of a 1 GiB instance's
full allocation - against a total of 5 worker requests in that entire window.

There is no tail. The container simply never sleeps, which is exactly what the
SIGTERM diagnosis in #154 predicts: `@cloudflare/containers` sends SIGTERM and
never escalates, node is PID 1 and ignored it, so the instance only ever goes
away when a deploy replaces the placement.

Two claims in CLAUDE.md were wrong as a result, and both were cost claims:

- "That cost ~$28/month for weeks" implied the drain fix ended it. Billing says
  the instance ran at 100% of a 4 GiB day on EVERY full day from 2026-07-20
  through 2026-08-03 (95.3 - 99.0 GiB-hours/day against the 96.0 a 4 GiB
  instance bills for 24h), including the five days after that fix deployed on
  2026-07-29. The bill was still being paid; the 2026-08-03 downsize to `basic`
  cut it ~4x rather than ending it. The drain guard stays - the hazard is real -
  but it was never the load-bearing cause.
- The disk-to-memory ratios "2.0" and "4.0" are dashboard units. Queried in
  bytes, which is what the GraphQL dataset returns, they read 1.86 and 3.73.

Adds the SIGTERM cause and the backfill trap as their own bullets, so the next
person reading absence-of-bucket as sleep has the rule in front of them.

Measured via the `containersUsageAdaptiveGroups` and `workersInvocationsAdaptive`
GraphQL datasets on account 3e467e5d..., 2026-08-05.

NOT a behaviour change: the fix itself is 343df53, already on main and NOT yet
deployed. Production still runs worker version 2e67470f and container app
version 3 (image sha256:16205eda..., updated 2026-08-03T21:24:01Z), both of
which predate the handler.

Verified: pnpm run verify green, exit 0 - 1588 app tests passed / 3 skipped,
13 worker, 4 container (including "exits promptly on SIGTERM"), 0 warnings.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01QZvNS2H4cbaZk46ybA7hbj
@wormeyman
wormeyman merged commit 901e832 into main Aug 5, 2026
6 checks passed
@wormeyman
wormeyman deleted the container-never-sleeps-measurement branch August 5, 2026 04:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant