Skip to content

fdev host hygiene: browser-run concurrency and /tmp headroom in the capacity probe, disk-backed test temp, dead-run /tmp cleanup #5240

Closed

Description

@EHotwagner

Parent Outcome: #4730
Work level: Delivery · Track: Active delivery · Intended Status: In progress

Owner

Root/fdev assigns.

Done when

From the 2026-11-02 progress report, process problem 4. On fdev the quiet-host bench probe never admitted for 60 minutes (load 5–9), /tmp (a 16 GB tmpfs) filled and caused 38 spurious SDD test failures, and many concurrent browser smokes ran.

  1. scripts/fdev-lane-capacity.py (Process fixes from the 2026-11-01 progress report: load-ceiling lane admission, review-finding triage, push before stop #5172) holds when:

    • Browser runs: at least 2 browser-smoke runs are active. Classify a headless Chromium main process (Playwright or a CDP-driven chrome with --headless and --remote-debugging-*), not its renderer or GPU children.
    • /tmp space: free space on /tmp is below a threshold, default 3 GiB, reported separately from disk.

    Both are configurable, with fixture tests in the existing style.

  2. Disk-backed test temp by default: ~/.local/bin/fdev-chromium-env, and documented guidance for .NET test runs, set TMPDIR to a disk path (for example ~/.cache/fsgg-tmp) unless already set. Keep it short enough for Unix-socket paths (the SDD socket test failed under a long TMPDIR). Document it in the fdev ops doc.

  3. Dead-run cleanup, one owner: a script removes /tmp and TMPDIR entries matching known harness patterns (fsgg-sdd-tests-<pid>-*, fsgg-gate-*, Playwright profile dirs) only when their PID is dead and they are older than 1 h, with a dry-run mode. It runs from the existing board-tend user units (fdev: board-tend user units, post-merge trigger, empty enable file and 7-day observation (Refs #5136) #5205) on a timer. Never delete a live run's files.

  4. Tests for the probe additions and the cleanup matcher (a fixture tree; live-PID dirs are kept).

Current deliverable

None.

Next action

Dispatch.

Blocked on

Nothing.

Acceptance evidence

None yet.

Refs #4730, Refs #5172

Activity

  1. EHotwagner commented on Nov 2, 2026

    @EHotwagner
    ContributorAuthor

    Root/fdev install of the merged #5242 (acfe0b8d) on fdev, following docs/coordination/fdev-host-hygiene.md:

    • Backup: the old ~/.local/bin/fdev-chromium-env is kept as fdev-chromium-env.pre-5240.
    • Wrapper: installed (sha256 8cbc5e15…). fdev-chromium-env --print-tmpdir prints /home/developer/.cache/fsgg-tmp.
    • Cleanup: fdev-tmp-cleanup.py (sha256 6fcb5fe6…) plus the fsgg-tmp-cleanup.{service,timer} user units. The timer is enabled; its first run is 2026-11-02 08:38 CET, then hourly.
    • Pre-enable dry run: 1077 matched, 880 would be removed (about 3.6 GB), apply_blocked: false, 0 unreadable processes.
    • Capacity probe live: admit, browsers 0, /tmp free 6.15 GiB.

    Acceptance pending: the first timer run's journal line. Root closes this after reading it back.

  2. EHotwagner commented on Nov 2, 2026

    @EHotwagner
    ContributorAuthor

    Root/fdev acceptance. The first live timer run (2026-11-02 08:38 CET) of fsgg-tmp-cleanup.service ended with Result=success and exit 0. Its JSON line:

    • mode: apply, apply_blocked: false, 367 processes read;
    • 1080 matched and 901 removed, failed: 0;
    • freed 3,619,803,136 bytes.

    /tmp went from 61% to 40% used (6.2 GB used, 9.6 GB free).

    Every Done-when item is met:

    1. The probe holds at ≥2 browsers and at /tmp under 3 GiB, verified live in review.
    2. Disk-backed TMPDIR via fdev-chromium-env, installed.
    3. Dead-run cleanup: one owner and an hourly timer, live and protected (independently reviewed with E1–E9 holder experiments).
    4. Tests: 37, 30, 13 and 43.

    Closing as completed. The non-blocking review notes are recorded in #5242's merge comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions