capsize — local test fixture

A four-node kind cluster that is broken on purpose, so capsize has something real to find.

Costs nothing. Runs on your laptop. No AWS account, no credentials, no cloud spend.

make up      # create the cluster (~90s)
make load    # apply the broken-on-purpose workloads
make verify  # show the inputs blast radius is computed from
make down    # tear it all down

make load waits for every deployment to become Available, then each container allocates a fixed amount of resident memory (see the table below) so metrics-server reports believable usage.

Wait for the settle window before trusting usage

make load returning does not mean the fixture is ready. Available means the pods are running, not that they have finished allocating. There is a window afterwards, typically 30–60 seconds, in which the fixture lies to you in two different directions:

• Memory is still climbing. The largest ballast is 768Mi and takes seconds to fault in. A tool that scans mid-allocation sees a workload as over-provisioned that is not.
• CPU is still busy. The dd burns real CPU while it runs. Measured on a GitHub runner mid-window, prod/good-api reported 9m and sandbox/limits-only 33m — enough for good-api, which requests 100m, to look CPU-over-provisioned. good-api is one of the two false-positive controls, so that turns the most important assertion in the table below into a coin flip.

So wait for both, not just memory. The check capsize's own end-to-end job uses:

# memory up AND cpu idle, held across two consecutive polls
stable=0
for i in $(seq 1 60); do
  top=$(kubectl top pods -A --no-headers 2>/dev/null || true)
  mem=$(echo "$top" | awk '$1=="sandbox" && $2 ~ /^limits-only-/ {gsub(/Mi/,"",$4); print $4+0}')
  zero_mem=$(echo "$top" | awk '$1 ~ /^(prod|sandbox|batch)$/ && $4 == "0Mi"' | wc -l)
  busy_cpu=$(echo "$top" | awk '$1 ~ /^(prod|sandbox|batch)$/ && $3 != "0m"' | wc -l)
  if [ -n "$mem" ] && [ "$mem" -ge 700 ] && [ "$zero_mem" -eq 0 ] && [ "$busy_cpu" -eq 0 ]; then
    stable=$((stable + 1)); [ "$stable" -ge 2 ] && break
  else
    stable=0
  fi
  sleep 10
done

Two consecutive clean polls, not one: a single quiet scrape during the allocation reads as settled when it is not. This repository's smoke job runs exactly this check, so if it stops being reachable the CI goes red here rather than surfacing as flake in whatever is consuming the fixture.

metrics-server is not installed for you

The cluster comes up without it, and without it capsize's request-versus-usage rules cannot fire — you still get every static finding. To install it:

kubectl apply -f vendor/metrics-server/components.yaml
kubectl patch deploy metrics-server -n kube-system --type=json \
  -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
kubectl wait --for=condition=Available deploy/metrics-server -n kube-system --timeout=120s

The --kubelet-insecure-tls patch is needed because kind's kubelet certificates are self-signed.

That is the vendored v0.9.0 manifest, which is what CI applies too — so a local run and a CI run are looking at the same metrics-server. Pulling releases/latest/download/components.yaml instead would make them different clusters on any day upstream has cut a release, and the difference would show up as a fixture that behaves one way on your laptop and another in CI. See vendor/metrics-server/README.md.

---

The node topology

Node	instance-type label	Pool	Capacity	Why it exists
control-plane	—	—	—	tainted, nothing schedules here
worker (1)	m5.large	ondemand-small	ON_DEMAND	tight allocatable — small denominators, high ceiling ratios
worker (2)	m5.2xlarge	ondemand-large	ON_DEMAND	roomy, on-demand — the "safe" node
worker (3)	m5.2xlarge	spot	SPOT	the node that makes an unbounded workload lethal

Instance types are labels, which is all a pricing join needs. No bill required, ever.

Allocatable memory is differentiated with kubelet system-reserved so the three workers genuinely have different capacity and the blast-radius denominator varies. Both eks.amazonaws.com/capacityType and karpenter.sh/capacity-type are set, since real clusters use one or the other; the spot node also carries node.kubernetes.io/lifecycle.

---

Expected findings — this table is the test oracle

If capsize's output disagrees with this table, one of them is wrong. Find out which.

Every column below was read off the manifests and confirmed against a live run. "Ballast" is the resident memory each container allocates into a tmpfs emptyDir, which counts against the container's cgroup and so shows up in kubectl top.

Workload	ns	Node pool	Replicas	Requests	Limits	Ballast	req/usage	Should be flagged as
good-api	prod	ondemand-large	2	128Mi / 100m	256Mi / 500m	96Mi	1.3x	⭐ NOTHING. A finding here is a false positive.
legacy-worker	sandbox	ondemand-small	1	256Mi / 200m	—	160Mi	1.6x	missing limits, and nothing else
orphan-job	sandbox	ondemand-small	1	—	—	32Mi	n/a	no requests and no limits
limits-only	sandbox	ondemand-large	1	—	1Gi / 1	768Mi	1.3x	requests left implicit, on both resources
overprovisioned-cache	sandbox	ondemand-large	1	512Mi / 1	—	40Mi	12.8x	over-provisioned + missing limits + the contradiction. See below.
spot-tenant-a/b/c	batch	spot	2 each	64Mi	—	48Mi	1.3x	missing limits; highest blast radius of any fixture workload
memory-balloon	batch	spot	1	64Mi	—	grows	—	opt-in only, not applied by make load

The four workloads sitting at 1.3–1.6x are deliberate. They are below any sensible divergence threshold, so each demonstrates exactly one defect and a usage-based rule firing on them is a false positive.

Namespaces

Namespace	Has LimitRange	Has ResourceQuota	Should be flagged
prod	yes (defaults)	yes (prod-quota)	⭐ NO. Another false-positive check.
sandbox	no	no	yes
batch	no	no	yes

A real cluster will also flag default, kube-node-lease, kube-public, kube-system and local-path-storage, none of which this fixture creates. That is correct behaviour, not a fixture defect.

limits-only is subtler than it looks

It declares limits and no requests. The Kubernetes core API — not a LimitRange, no admission plugin — defaults the requests to equal the limits. The scheduler therefore reserves the full 1Gi and the pod is Guaranteed, the least evictable QoS class, not BestEffort.

A tool that reads the PodTemplateSpec verbatim sees "no requests" and scores this workload as if nothing bounded it. This row exists to catch that.

⭐⭐ Why overprovisioned-cache is the whole product

It requests 512Mi and uses 40Mi. Every incumbent tool finds this and says "shrink your request, you're wasting 460Mi." They are right about the money.

It also has no limits.

Shrink its request and the scheduler packs more neighbours onto its node, while this workload remains free to consume the entire node ceiling. The cost recommendation makes the outage more likely.

capsize should emit the savings and the warning. Nothing else in the category emits the second half.

---

The ceiling term

ceiling     = min(node_allocatable_mem, container_mem_limit)
ratio       = ceiling / workload_mem_request
neighbours  = distinct other workloads schedulable on that node
spot_factor = 1.5 if the node is spot/preemptible else 1.0
risk        = ratio × log2(1 + neighbours) × spot_factor

With no limit set, the ceiling falls back to node allocatable, so an unbounded workload scores dramatically higher than an identical bounded one. That is the thesis expressed arithmetically: a limit is the thing that contains blast radius, and the metric rewards setting one automatically rather than through a special case.

Two workloads from this fixture, measured:

Workload	ceiling	request	ratio	neighbours	spot	risk
good-api	256Mi (its limit)	128Mi	2.00	4	1.0	4.64
spot-tenant-a	2.63Gi (the node — no limit)	64Mi	42.07	5	1.5	163.12

A 35x spread between two workloads of similar size, driven almost entirely by whether someone set a limit.

⚠️ Your absolute numbers will differ, and that is fine

The ceiling term is node allocatable memory, and on kind that comes from whatever RAM Docker Desktop (or your container runtime) was given, minus the system-reserved this fixture sets. The run above was on a host where the workers reported 644Mi and 2.63Gi allocatable. A machine with more RAM allotted to Docker produces a larger ceiling, a larger ratio, and a larger risk score for every unbounded workload — while bounded workloads like good-api stay put at 2.00, because their ceiling is their own limit.

So do not compare your numbers to the ones above. Compare ratios: bounded versus unbounded, spot versus on-demand, crowded node versus quiet one. Those relationships hold on any host. Run make verify to see the allocatable your cluster actually has.

---

⚠️ Optional: watch it actually break

make balloon     # unbounded container grows until it starves worker-spot
kubectl get pods -A -w
make unballoon

Bounded to 1200M so it cannot take your laptop down, but it will evict its neighbours on the spot node. That is the demonstration, not a bug.

---

Limitations

Kept honest on purpose.

• kind nodes share host memory. "m5.2xlarge" is a label, not real sizing. Allocatable is simulated via system-reserved, so absolute numbers are fiction; the relative differences and the code paths are real. See the note above.
• Ballast is resident, not representative. Each container dds a fixed block into a tmpfs emptyDir and then sleeps. That gives metrics-server something believable to report and exercises request-versus-usage rules honestly, but it is a flat line — it does not exercise sizing against a realistic usage curve with peaks.
• CPU usage is ~0 only once settled. The containers sleep after allocating, but the dd burns CPU on the way there — see the settle window above. Steady-state CPU is a flat ~0, which exercises the absence of CPU pressure and not much else: memory findings are exercised properly, CPU findings are not.
• limits-only needs room to roll. It is a single replica pinned by nodeSelector with the default maxUnavailable: 0, so on a re-apply the old pod holds the only 1Gi reservation and the new one cannot schedule. If it sits Pending, kubectl scale deploy/limits-only -n sandbox --replicas=0 and back to 1.
• No AWS is contacted. Any pricing join runs off fabricated node labels.

---

Contributing

make init       # step 1 in any clone: installs the pre-push gate
make test-hook  # prove the gate rejects and accepts what it claims
make identity   # run the gate over all of this repository's history

core.hooksPath is per-clone configuration and does not travel with a clone, so the same rules run server-side in the identity CI job.

Two rules, enforced over every commit in a push range and over all history in CI:

• One canonical identity, author and committer, on every commit. Annotated tags are checked the same way — the tagger must match too.

• An allowlist on trailers. Only three keys may appear in a commit's trailer block, or in an annotated tag's body; every other key is refused:

  trailer	rule
  Signed-off-by	must be exactly Paul Bezilla <[email protected]>
  Verified	free text
  Measured	free text

This replaced a scan for a list of vendor and tool names. Measured before it was removed: across the full history of all six repositories in this family, 207 commits, that scan matched nothing. A denylist only catches what somebody already thought to write down and is stale the day a new tool ships; an allowlist refuses an unlisted key whether or not the gate has heard of what wrote it.

The trailer rule has one sharp edge

Whether a Key: Value line is a trailer depends on which paragraph it lands in. git parses only the last paragraph, and only when the whole paragraph parses as trailers:

Add a thing                          Add a thing

Verified: 3 runs, 0 failures.        Verified: 3 runs, 0 failures.

And a closing paragraph.             ← nothing after it

The left-hand message ends in prose, so Verified: there is ordinary text the gate never looks at. The right-hand one ends with that line, so it is a trailer and its key must be allowlisted. Same words, two outcomes, decided by what comes after.

That is git's own definition, read with git interpret-trailers --parse, and it is the definition the tools that stamp provenance use. A ^Key: regex would be simpler and would reject ordinary prose — across the six repositories there are 53 distinct Key: Value shapes that are not trailers, one of them in this repository.

If a push is refused for a trailer you thought was prose, check whether it ended up last. A new evidence word needs adding to the allowlist before it can land there.

What the allowlist does not catch, on purpose

Two things pass this gate that an earlier version of it would have stopped. Both are the deliberate reduction, not an oversight.

A vendor or tool name in the body of a message. The allowlist reads the trailer block and nothing else, so such a name written in a paragraph of prose is ordinary text and is accepted. Attribution is stamped as a trailer, and an unlisted key is refused whether or not the gate has heard of the tool that wrote it — a stronger guarantee than a name list can give, because it does not need updating when a new tool ships. Matching words in prose is a different job, and the denylist that did it matched nothing across the full history of every repository in this family.

Anything in the working tree. Nothing greps the checkout for vendor names. Hand-written hooks under .git/hooks/ once did, and core.hooksPath makes git ignore that directory entirely, so any that survive there are inert. They have not been restored and should not be: it is the same scan with the same zero matches, and it walked build artefacts, so a full validation run could leave a clean tree unpushable.

The same trade is taken in every repository that shares this gate. Consistency across them is the property worth keeping — a one-repository exception would be the defect, not the fix.

History was not rewritten when this changed. No force push, no retag, nothing dropped.

Related

github.com/bezilla/capsize — the read-only Kubernetes CLI this fixture exists to test.