session: bound a job subtree's CPU and process count

docs/changes/unreleased/1190-cpu-bound.md

@@ -0,0 +1,52 @@
+---
+kind: added
+title: a job subtree is cut when it passes its CPU or process bound
+pr: 1190
+surface: [chat, engine]
+invalidates:
+  - "A run's only bounds were its dollar cap and, since #1158, its memory. Nothing bounded a job subtree's CPU or its process count, so a model that spawned busy loops as bash jobs (`yes`, `awk 'BEGIN{for(;;){}}'`) spent no tokens and no bound ever fired. A job subtree is now cut when it passes its bound, and the run is told why."
+---
+
+A job subtree is the process group #1155 already records at launch with its
+leader's identity, plus every process under it. It now has a bound of its own: a
+Go-side watch, portable, sampling the subtree's cumulative CPU and its live
+process count through the group handle. The bound has two halves that apply in
+different places.
+
+**The process backstop applies always.** A subtree may not hold more than 128
+processes — about four times the measured honest peak of 34 — quota'd or not,
+because a fork storm that has not yet accumulated CPU is invisible to a rate and
+a cgroup CPU quota does nothing to a process count.
+
+**The CPU rate rule applies only where nothing else caps the process.** When a
+cgroup CPU quota or an affinity mask BINDS this process — the cores it may use
+are fewer than the machine's — the quota is already the ceiling on what any
+subtree under it can burn, and #1187's park bound is what returns a turn wedged
+on a job that never ends. A CPU rule there would only cut honest work early:
+inside a four-core cell an ordinary `go test` legitimately uses most of its four
+cores, and a share of them would end it. So under a binding quota the CPU rule is
+OFF and only the process backstop applies. When NOTHING binds — the process may
+use the whole machine — the CPU rule is the only thing between a spinner storm
+and the box: a subtree may not sustain more than three fifths of the machine's
+cores. On the 20-core box the numbers came from that is 12 cores, clearing the
+honest peak (8.5) and cutting the storm (16 loops → 16 cores, 64 → 20).
+
+Whether a quota binds is read from the process's own cgroup CPU quota (v2
+`cpu.max`, v1 `cpu.cfs_quota_us`/`cpu.cfs_period_us`, walked from the process's
+cgroup to the hierarchy root the way the memory bound is read, #1162) and its
+scheduler affinity mask; off Linux nothing binds and the machine-cores rule
+governs as before.
+
+**Sustained over half a minute, not a spike.** A build burst can pass twelve
+cores for several seconds, so the bound trips only after the subtree has been
+over on fifteen consecutive readings at a two-second interval — thirty seconds
+sustained. A transient never cuts, and a storm — over on every reading by
+construction — is cut half a minute in. A subtree whose usage cannot be read
+(off Linux, or a group whose identity no longer matches) is never cut: silence is
+not pressure.
+
+**The run learns it, and it is not a silent kill.** A subtree that passes its
+bound is ended and the run is handed the record `the job subtree was not settled
+within its bound`, naming the job and what it was holding — #1183's and #1187's
+family, said about the subtree. The kill is a requested death, so the registry's
+own ending says nothing and this record is the only account of it.

internal/processgroup/cores.go

@@ -0,0 +1,35 @@
+package processgroup
+
+import "runtime"
+
+// EffectiveCores is how many processor cores this process may ACTUALLY use: the
+// smallest of the machine's logical CPUs, the scheduler affinity mask, and the
+// cgroup CPU quota walked from the process's own cgroup upward. A job subtree's
+// CPU ceiling is a share of THIS and not of the machine's raw core count,
+// because where a job runs the subtree is usually held under a cgroup CPU quota
+// or an affinity mask and can never reach more cores than these; a ceiling drawn
+// from the machine's cores would sit above anything the quota lets the subtree
+// produce and would never trip.
+//
+// It is a variable so a test — or a caller that already knows the figure — can
+// stand a core count in front of the bound without a cgroup or an affinity mask
+// to be under; nothing but a test reassigns it.
+var EffectiveCores = effectiveCores
+
+// effectiveCores folds the three readable bounds into the tightest, never below
+// one. A bound that cannot be read is a zero that does not participate — the same
+// silence rule the memory reader keeps (#1162): an unread bound is not a licence
+// to guess a smaller machine.
+func effectiveCores() float64 {
+	cores := float64(runtime.NumCPU())
+	if quota := cgroupCPUCores(); quota > 0 && quota < cores {
+		cores = quota
+	}
+	if affinity := affinityCores(); affinity > 0 && float64(affinity) < cores {
+		cores = float64(affinity)
+	}
+	if cores < 1 {
+		cores = 1
+	}
+	return cores
+}

internal/processgroup/cores_linux.go

@@ -0,0 +1,157 @@
+//go:build linux
+
+package processgroup
+
+// cores_linux.go reads the two Linux bounds on how many cores this process may
+// use — the cgroup CPU quota and the scheduler affinity mask — mirroring the way
+// cmd/codeaf reads the cgroup MEMORY bound (#1162): the process's own
+// /proc/self/cgroup names its cgroup, and the quota is walked from that cgroup to
+// the hierarchy root with the tightest taken, because a parent slice can bound
+// tighter than the leaf.
+
+import (
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+
+	"golang.org/x/sys/unix"
+)
+
+// cgroupCPUCores is the CPU quota the cgroup this process runs in places on it,
+// in cores (quota / period), or zero when the hierarchy places no quota to give.
+func cgroupCPUCores() float64 {
+	return cgroupCPUCoresAt("/proc/self/cgroup", "/sys/fs/cgroup")
+}
+
+// cgroupCPUCoresAt is [cgroupCPUCores] against a named cgroup file and hierarchy
+// root, so the walk can be tested without being in a cgroup. Both generations are
+// read: v2 has an empty controller list on its 0:: line and the quota in
+// <path>/cpu.max ("quota period", or "max period" for no quota); v1 names cpu
+// among its controllers and keeps the quota in cpu/<path>/cpu.cfs_quota_us over
+// cpu.cfs_period_us (a quota of -1 is no quota).
+func cgroupCPUCoresAt(procSelfCgroup, root string) float64 {
+	raw, err := os.ReadFile(procSelfCgroup)
+	if err != nil {
+		return 0
+	}
+	best := 0.0
+	for _, line := range strings.Split(string(raw), "\n") {
+		fields := strings.SplitN(strings.TrimSpace(line), ":", 3)
+		if len(fields) != 3 {
+			continue
+		}
+		controllers, path := fields[1], fields[2]
+		var cores float64
+		switch {
+		case controllers == "":
+			cores = smallestCPUCores(root, path, cpuMaxV2)
+		case hasController(controllers, "cpu"):
+			cores = smallestCPUCores(filepath.Join(root, "cpu"), path, cpuMaxV1)
+		default:
+			continue
+		}
+		if cores > 0 && (best == 0 || cores < best) {
+			best = cores
+		}
+	}
+	return best
+}
+
+// smallestCPUCores returns the tightest finite core quota that read names at the
+// cgroup directory dir or any ancestor up to root, or zero when none bound.
+func smallestCPUCores(root, dir string, read func(string) float64) float64 {
+	root = filepath.Clean(root)
+	at := filepath.Clean(filepath.Join(root, dir))
+	if at != root && !strings.HasPrefix(at, root+string(filepath.Separator)) {
+		return 0
+	}
+	best := 0.0
+	for {
+		if cores := read(at); cores > 0 {
+			if best == 0 || cores < best {
+				best = cores
+			}
+		}
+		if at == root {
+			break
+		}
+		parent := filepath.Dir(at)
+		if parent == at {
+			break
+		}
+		at = parent
+	}
+	return best
+}
+
+// cpuMaxV2 reads a cgroup v2 cpu.max at dir and returns quota / period in cores,
+// or zero when the file is absent, unreadable, or says "max".
+func cpuMaxV2(dir string) float64 {
+	raw, err := os.ReadFile(filepath.Join(dir, "cpu.max"))
+	if err != nil {
+		return 0
+	}
+	fields := strings.Fields(string(raw))
+	if len(fields) != 2 || fields[0] == "max" {
+		return 0
+	}
+	quota, err := strconv.ParseFloat(fields[0], 64)
+	if err != nil || quota <= 0 {
+		return 0
+	}
+	period, err := strconv.ParseFloat(fields[1], 64)
+	if err != nil || period <= 0 {
+		return 0
+	}
+	return quota / period
+}
+
+// cpuMaxV1 reads a cgroup v1 cpu.cfs_quota_us over cpu.cfs_period_us at dir and
+// returns their ratio in cores, or zero when there is no quota (the file is
+// absent or holds -1).
+func cpuMaxV1(dir string) float64 {
+	quota := readInt(filepath.Join(dir, "cpu.cfs_quota_us"))
+	if quota <= 0 {
+		return 0
+	}
+	period := readInt(filepath.Join(dir, "cpu.cfs_period_us"))
+	if period <= 0 {
+		return 0
+	}
+	return float64(quota) / float64(period)
+}
+
+// readInt reads one integer from a cgroup file, or zero when it cannot.
+func readInt(path string) int64 {
+	raw, err := os.ReadFile(path)
+	if err != nil {
+		return 0
+	}
+	n, err := strconv.ParseInt(strings.TrimSpace(string(raw)), 10, 64)
+	if err != nil {
+		return 0
+	}
+	return n
+}
+
+// hasController reports whether a cgroup v1 controller list (comma-separated)
+// names want.
+func hasController(list, want string) bool {
+	for _, c := range strings.Split(list, ",") {
+		if c == want {
+			return true
+		}
+	}
+	return false
+}
+
+// affinityCores is how many CPUs the process's scheduler affinity mask allows,
+// or zero when it cannot be read.
+func affinityCores() int {
+	var set unix.CPUSet
+	if err := unix.SchedGetaffinity(0, &set); err != nil {
+		return 0
+	}
+	return set.Count()
+}

internal/processgroup/cores_other.go

@@ -0,0 +1,10 @@
+//go:build !linux
+
+package processgroup
+
+// Off Linux there is no /proc/self/cgroup and no sched_getaffinity to read here,
+// so neither bound participates and [effectiveCores] falls back to the machine's
+// logical CPU count.
+func cgroupCPUCores() float64 { return 0 }
+
+func affinityCores() int { return 0 }

internal/processgroup/usage.go

@@ -0,0 +1,33 @@
+package processgroup
+
+// A job subtree is a process group, and a process group is addressable — #1155
+// recorded every group at launch with its leader's identity precisely so it
+// could be named later. That same handle is what a bound on a subtree's
+// resource use needs: the leader's pid is the root of the tree, and the tree is
+// exactly what the job started.
+//
+// WHAT THIS MEASURES, AND IN WHAT UNIT. Two things, because a bound in this
+// code is stated in cores and processes and both have to be readable together:
+//
+//   - Processes is how many processes are alive in the subtree at this instant.
+//   - CPUSeconds is the subtree's CUMULATIVE processor time (user plus system)
+//     over every process in it. A cumulative figure is what a RATE is taken
+//     from: the difference of two readings over the wall between them is the
+//     cores the subtree held, which is the reading this product bounds. One
+//     reading on its own is not a rate and must not be read as one.
+//
+// THE READING IS /proc, WHICH IS LINUX'S. A platform without it answers "cannot
+// say" rather than a guess, and a bound that cannot say never cuts — the same
+// silence rule the admission governor keeps (session's task_pressure.go): a
+// person on a machine this cannot measure gets exactly the scheduler they had
+// before the bound existed.
+
+// Usage is what one job subtree is using at one instant.
+type Usage struct {
+	// Processes is how many processes are alive in the subtree.
+	Processes int
+	// CPUSeconds is the subtree's cumulative CPU time in seconds, summed over
+	// every process in it. Subtract two readings and divide by the wall between
+	// them to get the cores the subtree held.
+	CPUSeconds float64
+}