Skip to content

0.42: DSH timings, cached input tokens, diff-edits restriction, leaderboard and turns-nudge - #79

Merged
MathisWellmann merged 7 commits into
mainfrom
misc-improvements
Sep 24, 2026
Merged

0.42: DSH timings, cached input tokens, diff-edits restriction, leaderboard and turns-nudge#79
MathisWellmann merged 7 commits into
mainfrom
misc-improvements

Conversation

@MathisWellmann

@MathisWellmann MathisWellmann commented Sep 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Breaking changes since 0.41:
- AgentRun and PartialRun carry a new public timings field.
- RevisionToolError has new variants EditRequired and Stopped.
- build_revision refuses complete candidates after the first registered
build; later changes go through edit_revision.
- The EvaluateRevisionTool closure returns Result<E, String> with
E: Into; String still converts.

@MathisWellmann
dsh: measure per-request and per-tool timing and export it
d6226be 
The trace timed an agent run only as a whole, so the DSH export split
the run's model time evenly over its turns. A 10 s tool execution then
looked like slow model calls, and model latency could not be told apart
from tool time.

The `Recorder` hook now also keeps the run's clock: when each completion
request was sent and answered, and when each tool call started and
finished (by the call id the transcript uses). `AgentRun`, `PartialRun`
and `RunTrace` carry it as `RunTimings`, re-based onto the attempt's
start so the wait for an inference slot is not counted as model time.
The export places `step/start`, `assistant/message` and `tool/result`
on the measured instants and keeps the even split as the fallback for
traces without timings (serde default, so old traces still load).
@MathisWellmann
dsh: export cached input tokens
cf488c5 
The export dropped rig's cache counts because DSH counts are disjoint
(billed input = input + cache read + cache write) while rig leaves it to
the provider whether `input_tokens` contains the cached tokens. A lane
served ~97% from sglang's radix cache therefore exported as if nothing
was cached.

Take the cache out of the input where the provider folded it in and
report it as `cacheReadTokens` / `cacheWriteTokens`. The total tells the
conventions apart: OpenAI-compatible servers total prompt + completion,
rig's Anthropic mapping adds the cache counts on top of an uncached
input. Zero cache counts stay absent (unreported, not 'no hits').
@MathisWellmann
tools: only diff edits after the first registered build
4f02619 
The v0.34 sliding traces show lanes re-sending the whole program for
every variant: 126 full build_revision calls against 55 edit_revision
diffs, successive full candidates a median 97% identical, and those
re-emissions two thirds of all output tokens a lane generated.

Once a lane has registered a revision, both tools refuse a complete
candidate: build_revision outright, edit_revision when its text carries
no edit at all (the whole-candidate fallback). The refusal
(RevisionToolError::EditRequired) names the registered revisions and the
base to edit, costs no parse and no build, and the tool descriptions
and the Revision tools prompt section state the rule.
@MathisWellmann MathisWellmann self-assigned this Sep 24, 2026
@MathisWellmann MathisWellmann changed the title DSH timings, cached input tokens, diff-edits restriction, leaderboard and turns-nudge 0.42: DSH timings, cached input tokens, diff-edits restriction, leaderboard and turns-nudge Sep 24, 2026
@github-actions

github-actions Bot commented Sep 24, 2026

Copy link
Copy Markdown

Summary

The PR adds measured per-request/per-tool run timings to traces and the DSH export, a with_code_deadline that halts runs spinning on documentation, cache-aware token accounting, a scored-evaluation stopping rule with a leaderboard, and post-registration refusal of complete candidates. The change is careful and well-tested; I found one major error-handling inconsistency in the new stopping rule and a coverage gap in its terminal state.

Issues

  • Major — symbiont/src/tools/build.rs:113: once the stopping rule ends the search, build_revision reports EditRequired ("change one of them with edit_revision") instead of Stopped, because require_edit() runs before reserve_build() and a stopped lane always has a non-empty built. The agent gets contradictory terminal guidance and the new "stopped" record_refused outcome (pipeline.rs:190) is unreachable from this tool. Check the stopped state before require_edit so both build tools surface the same refusal.
  • Major — missing test, symbiont/tests/revision_tools.rs:381: the stopping rule's terminal state has no integration coverage — with_stopping_rule(3) is installed but the scripted run never fires it, so the Stopped refusal through build_through_tool/reserve_build and the "builds nothing more" contract after it are untested end-to-end (only ToolContext-level tests exist). Add a scripted lane that evaluates patience+ non-improving revisions and asserts the refusal, the "stopped" trace outcome, and that submitting the best still works.

@MathisWellmann
tools: evaluate_revision leaderboard and stopping rule
4a6b660 
A fifth of the v0.34 lanes' model time went into revisions built after
the one they finally submitted, every one of them worse: lanes kept
tuning constants past the point where evaluations stopped improving.

The host's evaluate closure may now return an `Evaluation` with a score
(higher is better) instead of plain text. Scored reports end with the
lane's leaderboard (first score per revision, the best built here
marked, the active revision listed for reference only). With
`.with_stopping_rule(patience)` the lane stops after `patience` scored
revisions it built in a row did not beat its best: the answer says so
and the build tools refuse further candidates
(RevisionToolError::Stopped), naming the revision to submit. Plain-text
closures keep working and never stop a lane.

A stopped lane gets the same Stopped refusal from build_revision and
from a complete candidate sent to edit_revision: the stopped check runs
before the edit-only check, which would otherwise tell the lane to edit
a revision instead of submitting its best. Both refusals made before the
pipeline ("stopped", "edit_required") are recorded in the trace like
the budget refusals.
@MathisWellmann
runtime: nudge a run that writes no code for too many turns
ec133f6 
`Agent::with_code_deadline(turns)` stops a run that spends `turns`
tool-call turns in a row without calling `build_revision` or
`edit_revision`. The recorder hook counts the requests since the run
last wrote code and stops the next request with a `PromptCancelled`
whose reason names the deadline.

The runtime treats that stop like `MaxTurnsError`: it withdraws the
tools for the rest of the lane, sends the same nudge (naming the turns
spent), and reports the failure as `max_turns`. A run that only reads
documentation hears the nudge early; a run that keeps editing can still
use the whole `default_max_turns` budget. Without a deadline nothing
changes.
@MathisWellmann
0.42: measured DSH timings, cached tokens, edit-only revisions, stopp…
9cf7d4e 
…ing rule, code deadline

Breaking changes since 0.41:
- AgentRun and PartialRun carry a new public `timings` field.
- RevisionToolError has new variants EditRequired and Stopped.
- build_revision refuses complete candidates after the first registered
  build; later changes go through edit_revision.
- The EvaluateRevisionTool closure returns Result<E, String> with
  E: Into<Evaluation>; String still converts.
@MathisWellmann
dsh-log: fix clippy findings
f528847 
- types.rs: serialize the test line by value; to_value takes it by value.
- tests/v3_session.rs: give the crate-level allow its reason (denied by
  allow_attributes_without_reason) and expect large_enum_variant and
  too_many_lines on the fixture enum and the end-to-end session test.
@github-actions

github-actions Bot commented Sep 24, 2026

Copy link
Copy Markdown

Summary

The PR adds measured per-request/per-tool run timings that drive the dsh session export, a code deadline that stops doc-only agent runs, and an "edit-only after first registration plus patience stopping rule" for the revision tools. The design is mostly coherent, but the change as written does not compile, and the failure path misses the re-anchoring the success path performs.

Issues

  • Blocker — symbiont-dsh-log/src/types.rs:1049: serde_json::to_value(LogLine::SystemMessage(event)) passes the value by value, but to_value takes &T, so the test module no longer compiles and cargo test for symbiont-dsh-log (and any workspace-wide test run) fails. Restore the dropped &.
  • Major — symbiont/src/runtime.rs:627: the run-failure path stores timings: partial.timings without anchored(t_attempt_std), while the success path (line 1575) anchors; failed runs' trace timings stay relative to the run's start, violating RunTimings' documented attempt-relative invariant, so the attempt-to-run gap (e.g. waiting for an inference slot) is attributed inside the first request of the exported log. Anchor the failure path too.

@MathisWellmann
MathisWellmann merged commit 9af41c6 into main Sep 24, 2026
5 of 6 checks passed
@MathisWellmann
MathisWellmann deleted the misc-improvements branch September 24, 2026 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@MathisWellmann MathisWellmann

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@MathisWellmann