Skip to content

Harbor design page and contributor checklists; scrub a CLI's live stream, refuse multi-step tasks as unsupported - #119

Merged
haizhongzheng merged 3 commits into
mainfrom
docs-contributor-checklists
Nov 20, 2026
Merged

haizhongzheng merged 3 commits into
mainfrom
docs-contributor-checklists

Conversation

@haizhongzheng

@haizhongzheng haizhongzheng commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Change

After the task-environment refactor (#117), the contributor checklists and the task tutorial still described the
framework before it, and the Harbor design was only in the long prose of docs/framework/task.md#benchmarks.

  • New page, docs/developer/harbor-design.md ("How Harbor runs a task"): the processes (Hermit, the driver
    under Harbor's python, the task's container); who does each step of a trial; a trial in order with a command's path
    through the bridge; CLI agents versus loop agents (three inline SVG diagrams, docs/_includes/harbor-*.html, styled
    by docs/_static/css/harbor-diagrams.css for both themes, as the sandbox diagrams are); how the agent phase ends; outcomes and retries; why Harbor's
    Trial.run() owns the trial. Linked from docs/index.md, docs/developer/index.md and task.md#benchmarks,
    which stays the contract.
  • Sandbox checklist: a process may run in an environment the framework gives (hermit.environment: Local,
    Sandboxed, Container, a task's Harbor environment), not only through hermit.sandbox / hermit/isolate.py;
    a Harbor task's trial directory counts as its launch record; hermit/environment/ joins the files whose change is
    a core change.
  • New benchmark checklist (.agents/skills/prepare-pr/references/benchmark-checklist.md): task directories in
    Harbor's format, the public and private halves, what each task asks of its environment, parity for a moved
    benchmark, a real run. prepare-pr/SKILL.md and CONTRIBUTING.md say when it applies.
  • Task tutorial (docs/tutorials/your-own-task.md) and the developer index's "New task" row point to task
    directories and Benchmarks.
  • .agents/skills/dual-review: the independent reviewer is Codex gpt-6.2-sol (was gpt-6.1-sol).
  • Two driver fixes (hermit/environment/harbor_driver.py, found in the review, included at the maintainer's
    request; CHANGELOG under Fixed):
    • A CLI's output kept in /logs/agent as it ran reached a shared verifier unscrubbed when Harbor's agent timeout
      ended the phase before Hermit's own logs were copied over it. The phase's end now scrubs each such stream of the
      credentials passed to the agent's commands, in a local Docker environment (whose /logs/agent is the trial's
      folder on this machine): read without following a link, scrubbed before any copy is written, put back through
      safe_copy only when changed. It is best effort: a stream that cannot be read or replaced is logged and left as
      it came (as on main), never a failure of the phase's end. Elsewhere a shared verifier still gets it as it came.
    • A multi-step task ([[steps]]) was refused as sandbox_failed and run again after infra_wait_s; it is now
      refused as unsupported_task and never run again.
  • Merged main (Remove the forwards of moved modules #121) into the branch; no conflict.

Validation

  • bash docs/build.sh (Sphinx, warnings as errors): passes; the three diagrams checked in light and dark (screenshots).
  • Every claim of the new page was checked against hermit/environment/harbor.py, harbor_driver.py (the JSON-lines
    protocol, setsid and process groups, one transfer at a time, the end of the phase: any failure there is infra),
    harbor_errors.py and task.md#benchmarks (the 60 s margin, setup_timeout_sec 1200 s, infra_retries 2 after
    300 s, busy_retries 3, unscored runs).
  • python -m pytest --changed --base origin/main (every group: hermit/environment/ is shared) on a snapshot of the
    final tree: 1941 passed, 118 skipped, 0 failed (23 min).
  • New tests, each failing on main and passing here:
    test_a_clis_live_stream_is_scrubbed_before_the_verifier_even_when_harbors_timeout_ends_the_phase (the stream and
    its key registered through Driver.exec, Harbor's timeout without Hermit's logs, a planted link never followed, a
    non-mounted environment left alone, an unreadable stream never infra) and
    test_a_multi_step_harbor_task_is_refused_as_unsupported_and_never_run_again. Both run under Harbor's venv, like
    the other driver tests (skipped without it).
  • tests/core/test_harbor_environment.py: 115 passed; ruff check hermit/environment/harbor_driver.py: clean.

Review

  • Tests and documentation cover the affected behavior: the two new tests; the docs build.
  • Commands, defaults, and public interfaces are consistent: no code or config change; the numbers on the new page
    are the code's defaults (above).
  • Agents follow the agent layers: no agent changes; the new page describes agents through the Environment API.
  • Shared logs and examples contain no credentials or private data.
  • Documentation-only: not only documentation now; the driver fixes were asked for by the repository's maintainer (no issue).
  • Core changes: hermit/environment/harbor_driver.py (the two fixes above), a sandbox change by the sandbox
    checklist; agreed with the repository's maintainer, who asked for them. The checklists change what later PRs
    answer.
  • Implementation correctness and failures: the scrub never fails the phase's end (a failure is logged and
    the stream left as on main); a link is never followed; a partial staging write is never published; the
    multi-step refusal maps to unsupported_task, which is not retried.
  • Real execution paths and tests: the driver's own exec and _end_phase under Harbor's python; the docs build. Not run: a live Docker trial that times out with a key in the stream.
  • Documentation: the files listed under Change.
  • This description: read end to end against the diff.
  • Independent review (post-PR audit): /dual-review, this session and Codex gpt-6.2-sol, 9 rounds.
    Findings per round (Codex): 15, 13, 9, 6, 6, 5, 8, 3, 3; every one verified against the code. Fixed: the false
    claims in the page, diagrams and checklists, the two driver fixes, and regressions the scrub briefly introduced
    (made best effort). Left, under the maintainer's bar (fix only breakage, security, wrong results or cost, false
    statements): NUL-delimited streams, a host umask of 077, a mounts override of /logs/agent, a multi-step task
    that Modal compose rejects in Trial.create, a stalled filesystem past the phase-end watchdog, safe_copy
    leaving a partial file on ENOSPC (predates this PR), wording and accessibility nits.
  • Sandbox checklist: the driver runs no new process; the scrub is this machine's own code on the trial's folder.
    Test checklist: the new tests are in tests/core/, named as sentences, with docstrings saying why; each fails on
    main. Benchmark checklist: not applicable (no benchmark).

🤖 Generated with Claude Code

…refactor

After #117 the contributor checklists and the task tutorial still described
the framework before it. Add docs/developer/harbor-design.md with three
diagrams (the processes, a trial in order, CLI and loop agents) and who does
each step, how the agent phase ends and the outcomes; let the sandbox
checklist accept hermit.environment and count hermit/environment/ as core;
add a benchmark checklist (task directories, public and private halves, what
a task asks, parity, a real run) and say when it applies; point the task
tutorial and the developer index to task directories; the dual-review
reviewer is gpt-6.2-sol.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
haizhongzheng and others added 2 commits November 20, 2026 11:50
…rub a CLI's live stream; refuse multi-step tasks as unsupported

The independent review (/dual-review, 9 rounds) checked every claim of the
Harbor design page, its diagrams and the checklists against the code and
Harbor 0.25.0, and corrected what was false: Local is not confinement and
sandbox evidence must show the isolation probe passed; only the bubblewrap
task sandbox refuses a task the run cannot give, and every backend states its
effective timeouts; Harbor's trial order, what is infra, which retries apply,
what passes through the bridge, CLI install and reuse, the separate verifier.

Two driver fixes the maintainer asked to include:
- A CLI's output kept in /logs/agent as it ran reached a shared verifier
  unscrubbed when Harbor's timeout ended the phase before Hermit's logs were
  copied. The phase's end now scrubs each stream there, best effort, in a
  local Docker environment (never a failure of the phase's end).
- A multi-step task was refused as sandbox_failed and run again; it is now
  refused as unsupported_task and never run again.
Each has a test that fails on main.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@haizhongzheng haizhongzheng changed the title Docs: Harbor design page, sandbox and benchmark checklists after the refactor Harbor design page and contributor checklists; scrub a CLI's live stream, refuse multi-step tasks as unsupported Nov 20, 2026
@haizhongzheng
haizhongzheng merged commit 6940f26 into main Nov 20, 2026
1 check passed
@haizhongzheng
haizhongzheng deleted the docs-contributor-checklists branch November 20, 2026 19:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant