[Poncho1112]

In one operator's eval log, the claude engine recorded no tokens on any of 2,672 attempts. That left the biggest lane with no token or cost evidence on the scoreboard. Two problems stacked up:

1. No regex can read Claude Code's output. claude --print --output-format json prints a single JSON result line: {"type":"result", ..., "usage":{"input_tokens":..,"cache_creation_input_tokens":..,"cache_read_input_tokens":..,"output_tokens":.., ...}, "modelUsage":{...}}. Nested blocks later in that line also contain "output_tokens". parse_token_count takes the last match, which on a real result gave 606 instead of 4,745, and a regex can't add fields together anyway.
2. The configured regex was mis-escaped. In a single-quoted TOML string backslashes are literal, so '"output_tokens"\\s*:...' reached Python as a literal backslash followed by s and never matched anything.

What changes

• token_format = "claude-json" is a new optional engine key. It's inherited like token_regex, and unknown values are rejected. With it set, parse_claude_json_tokens() reads the top-level usage of the last result line and records non-cached input + cache creation + output tokens. Cache reads are left out because they re-read context that was already counted, and including them would inflate long sessions several times over.
• parse_worker_tokens() is used at the one call site that computes a worker attempt's tokens. If a claude-json engine prints no result line, it falls back to token_regex as before. Engines without token_format behave exactly as today.
• load_engines warns on stderr, without failing, when a token_regex has a doubled backslash before a regex class letter, the single-quoted-TOML mistake above.
• config.sample.toml gains a commented [engines.claude] example using the new key.

Tests

• tests/test_claude_tokens.py has 11 tests, all on synthetic data: nested output_tokens after the top-level usage, surrounding log lines, no result line, truncated JSON, last of two results wins, token_format loading and rejection, regex fallback, the mis-escape warning, and an end-to-end ringer.py run through a fake Claude-shaped engine that asserts the eval row's worker_tokens.
• RINGER_NO_SELF_UPDATE=1 python3 -m unittest discover -s tests passes all 295 tests.

This is independent of #154 (the INFRA verdict); either can merge first.

🤖 Generated with Claude Code