Add token-based review chunks, usage reporting, and longer timeout

README.md

@@ -110,7 +110,7 @@ Native API and subscription reviews return command output, exit status, and erro
 follow-up checks. Each command can run for up to 120 seconds and returns up to
 64 KiB of output, with truncation reported explicitly. File reads support paging.
 Each patch chunk allows up to 32 tool rounds or 128 tool calls, followed by a
-final review that records unfinished checks. A 10-minute deadline covers the
+final review that records unfinished checks. A 20-minute deadline covers the
 entire chunk; exceeding it fails the review.
 
 Other common scopes:
@@ -145,6 +145,59 @@ Repository-specific rules can be committed in `.diffvouch.yml`. DiffVouch reads
 that policy from the trusted base commit so a change cannot suppress its own
 review. A repository cannot select billable API transport or enable publishing.
 
+### Token limits and usage
+
+Diff chunks default to **50,000 tokens**, counted locally before any model call.
+Configure the limit in the trusted `.diffvouch.yml`:
+
+```yaml
+version: 1
+review:
+  chunk_tokens: 50000
+```
+
+Or override it for a review:
+
+```bash
+diffvouch review --provider codex --transport subscription --model gpt-5.6-sol --chunk-tokens 50000
+```
+
+This replaces the previous 180,000-byte default, not an exact conversion of it.
+Existing configurations with `chunk_bytes` keep their byte-based behavior.
+Choose one configuration field; `--chunk-tokens` overrides either. The separate
+`max_diff_bytes` collection safety limit remains in bytes.
+
+The tokenizer runs offline with bundled vocabularies. Recognized models use the
+encoding mapped by the tokenizer library. Unknown models, including newer model
+names and Claude, use `o200k_base` with an explicit estimate label. Counts are
+exact for that local encoding, not a guarantee of the selected model's accounting.
+
+Terminal output and the JSON `tokens` field report each chunk's diff and initial
+prompt text counts. Native OpenAI API and ChatGPT subscription reviews also
+record input, cached input, output, and reasoning tokens for each model request,
+when the provider returns usage. Progress goes to stderr, preserving JSON stdout.
+External CLI and Anthropic usage is currently unavailable. Missing usage is
+reported as unavailable, not zero.
+
+For context planning, use the **largest input count for a single request**, then
+leave room for reasoning and the answer. The cumulative input count across tool
+rounds measures usage and includes repeated history; it is not the context size.
+Cached tokens are already part of input, and reasoning tokens are already part
+of output. Do not add those subtotals twice.
+
+A chunk limit covers only the diff. Instructions, tool definitions, the output
+schema, and context gathered during review need additional room. For example,
+a chosen 128,000-token working budget might allocate 50,000 to the diff, 46,000
+to instructions and retrieved context, and 32,000 to reasoning and output. This
+is a planning example, not a measured model quality threshold. DiffVouch reports
+usage after requests; it does not yet enforce a total context budget or compact
+history. Establish the useful working budget through review-quality evals rather
+than assuming the model's maximum context is its best operating range.
+
+OpenAI documents the distinction between local text counts and full request
+counts in its [token-counting guide](https://developers.openai.com/api/docs/guides/token-counting),
+and explains context allocation in its [reasoning guide](https://developers.openai.com/api/docs/guides/reasoning).
+
 Secret redaction is temporarily disabled because altering source lines produced
 false findings. Review the diff for credentials before running DiffVouch: the
 complete reviewable patch is sent unchanged to the selected AI provider.

docs/eval-running.md

@@ -55,3 +55,18 @@ diffvouch eval compare --before /path/to/earlier/run --after /path/to/later/run
 ```
 
 A nonzero repeat selects the same deterministic approximately 20% subset for independent runs. `compare` requires the same manifest and writes observations on shared completed cases to `comparison.json`. Use it for code/prompt comparisons and repeat variation. Changes to the judge also change the measurement and should be assessed separately. No command enforces a regression threshold or CI quality gate.
+
+## Deferred task: evaluate chunk token limits
+
+Recorded 2026-09-21 at the user's request. Status: deferred; do not start until evaluations are explicitly resumed. Keep the current chunking implementation and configured value unchanged for now.
+
+Find the chunk token limit that gives the best review quality for the available subscription budget. Test sizes without changing how files or hunks are grouped, adding a synthesis pass, or modifying the prompt between size variants.
+
+- Freeze the reviewer implementation, the original prompt selected on 2026-09-30, model, effort, tools, judge/adjudicator settings, and case revisions. The completed prompt experiments remain separate from this deferred chunk-size experiment; do not attribute their differences to chunk size.
+- Preselect a representative development cohort before inspecting outcomes. Include enough PRs whose actual token counts exceed the smaller limits, large individual files, changes involving multiple files, and varied languages/repositories. Small PRs that remain a single chunk under every setting are a control, not evidence of an optimal limit. Expand the development collection if the existing 30 personal PRs lack enough large cases. Preserve untouched confirmation cases and keep benchmark labels and competitor output out of reviewer prompts.
+- Start with candidate limits of 20k, 35k, 50k, 75k, and 100k tokens, adjusted before the run if model/transport context constraints require it. Include a whole-diff condition only where sufficient context remains for instructions, tools, and reasoning/output. These are experiment values, not new defaults.
+- Before running, ensure the eval runner can pass an explicit chunk limit through to the reviewer and record it in the run identity/configuration. Record the tokenizer and whether local counts are estimates. Keep the byte collection safety limit and other per-chunk limits fixed, and record chunk count and actual usage. Different chunk counts can change the total available tool calls and review time; report that effect rather than treating total compute as equal.
+- Run paired cases for every setting, repeat a fixed subset to measure model variation, and retain failures and unfinished cases. Report reference coverage/precision/F2 where available, finding validity, concrete maintainability benefits, unresolved claims, completion rate, peak request input, cumulative usage, and elapsed time. Audit whether the prompt led to context inspection, execution of relevant existing tests/analyzers, candidate verification, and accurate reporting when evidence was unavailable. If saved output is insufficient to establish a behavior, mark it unknown rather than assume compliance.
+- Compare paired results with uncertainty and repository breakdowns. Select the development setting based on quality and usage, then confirm on untouched cases before recommending a default. Do not add regression enforcement. Retain the current default if the results do not establish a useful improvement.
+
+Use the native ChatGPT subscription transport, stop when its allowance is exhausted, and never fall back to billable API calls. Use archived/published competitor results only; do not run competitor products for this experiment.

docs/review-quality-improvements.md

@@ -1,10 +1,12 @@
 # Review quality improvements beyond prompt changes
 
-Research date: 2026-09-21. These are proposed changes; this note does not implement them or resume evaluations.
+Research date: 2026-09-21. The architectural options below are research proposals, not the current implementation plan. The original code observations refer to commit `86fb943`. The larger evaluation campaign remains paused; partial prompt experiments are reported separately.
 
 ## Recommendation
 
-Build a staged review pipeline around the existing model call. The first priority is deterministic context and candidate coverage. The second is evidence-based verification and PR-level synthesis. Static analysis, tests, semantic deduplication, and adaptive compute should feed those stages. Explicit orchestration can enforce context selection and review coverage that the current model-directed tool use does not guarantee.
+Decision on 2026-09-30: keep the original prompt. The experimental expanded prompt was removed from the working implementation at the user's request. Across the same eight completed personal PRs, the original produced 17 accepted claims, the installation-restricted variant produced 16 after the false-positive audit, and the corrected expanded variant produced 15 with one unresolved claim. These small, historical comparisons do not establish that either prompt is generally better. The supporting prompt evaluation and Tarpan retry reports remain local, uncommitted artifacts.
+
+The original instructions in [`buildPrompt`](../internal/review/review.go) permit repository inspection and relevant commands and tests when useful. The expanded workflow remains preserved in frozen experiment sources and reports. No architectural changes are selected here. The [deferred chunk-size experiment](eval-running.md#deferred-task-evaluate-chunk-token-limits) remains paused and will use the selected original prompt unless a later decision changes it.
 
 The ten-PR diagnostic supports this priority but does not prove it. Repository tools raised reference F2 from 41.4% to 47.2%, while the paired 95% interval was [-5.6, +15.2] points. The tool-enabled version still missed 17 of 32 benchmark issues, while the model judge marked only one of its 32 findings invalid. Unmatched claims are benchmark false positives even when the judge finds a real additional issue. These counts make coverage the first hypothesis to test. The small sample and model judge do not establish the cause, and they do not support blanket filtering. No version produced a valid maintainability finding. See [the paired report](eval-main-vs-tools-10.md#main-versus-repository-access-ten-prs).
 
@@ -17,7 +19,7 @@ Research gives useful design precedents, not a guaranteed score increase:
 
 First-party product accounts report similar structures, but they are not controlled comparisons and their quality claims should not be treated as proof. Cursor describes an early eight-pass, majority-vote, validator pipeline, then says its larger gains came from dynamic context and tool use. Anthropic describes parallel detection, verification, severity ranking, and compute that scales with PR complexity. These accounts justify testing those components separately. They do not justify copying a fixed number of passes or filtering every single-pass candidate. [Cursor Bugbot account](https://cursor.com/blog/building-bugbot), [Anthropic Code Review account](https://claude.com/blog/code-review)
 
-## What the current code does
+## What the original research baseline did
 
 - `gitdiff.ChunkPatch` fills byte-limited chunks with whole file sections, then individual hunks when a file is too large. It has no dependency or symbol boundary model. [internal/gitdiff/gitdiff.go lines 383-424](../internal/gitdiff/gitdiff.go#L383-L424)
 - `review.Perform` starts a fresh review for each chunk and concatenates the results. There is no final model pass that reasons across chunk findings or recovers a dependency split between chunks. [internal/review/review.go lines 111-123](../internal/review/review.go#L111-L123)
@@ -28,7 +30,7 @@ First-party product accounts report similar structures, but they are not control
 
 The paused full run `f003bd8b42da24a9` also shows why reliability needs its own workstream. Its local native records contain 81 completed cases and 30 errors: 19 checkout/dataset-patch mismatches, 9 incomplete subscription responses, 1 merge-base reconstruction failure, and 1 stream-read failure. Fixture preparation errors and provider transport errors should not be counted as review misses or mixed into an architectural quality conclusion.
 
-## Ranked implementation plan
+## Architectural options deferred for later evaluation
 
 | Rank | Change | Expected effect | Main risk and control |
 |---:|---|---|---|

go.mod

@@ -5,6 +5,7 @@ go 1.24.0
 require (
 	github.com/gofrs/flock v0.12.1
 	github.com/spf13/cobra v1.10.1
+	github.com/tiktoken-go/tokenizer v0.7.0
 	github.com/zalando/go-keyring v0.2.6
 	golang.org/x/term v0.35.0
 	gopkg.in/yaml.v3 v3.0.1
@@ -13,6 +14,7 @@ require (
 require (
 	al.essio.dev/pkg/shellescape v1.5.1 // indirect
 	github.com/danieljoos/wincred v1.2.2 // indirect
+	github.com/dlclark/regexp2 v1.11.5 // indirect
 	github.com/godbus/dbus/v5 v5.1.0 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/spf13/pflag v1.0.9 // indirect

go.sum

@@ -5,6 +5,8 @@ github.com/danieljoos/wincred v1.2.2 h1:774zMFJrqaeYCK2W57BgAem/MLi6mtSE47MB6BOJ
 github.com/danieljoos/wincred v1.2.2/go.mod h1:w7w4Utbrz8lqeMbDAK0lkNJUv5sAOkFi7nd/ogr0Uh8=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/dlclark/regexp2 v1.11.5 h1:Q/sSnsKerHeCkc/jSTNq1oCm7KiVgUMZRDUoRu0JQZQ=
+github.com/dlclark/regexp2 v1.11.5/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
 github.com/godbus/dbus/v5 v5.1.0 h1:4KLkAxT3aOY8Li4FRJe/KvhoNFFxo0m6fNuFUO8QJUk=
 github.com/godbus/dbus/v5 v5.1.0/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
 github.com/gofrs/flock v0.12.1 h1:MTLVXXHf8ekldpJk3AKicLij9MdwOWkZ+a/jHHZby9E=
@@ -24,6 +26,8 @@ github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY=
 github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
 github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
 github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
+github.com/tiktoken-go/tokenizer v0.7.0 h1:VMu6MPT0bXFDHr7UPh9uii7CNItVt3X9K90omxL54vw=
+github.com/tiktoken-go/tokenizer v0.7.0/go.mod h1:6UCYI/DtOallbmL7sSy30p6YQv60qNyU/4aVigPOx6w=
 github.com/zalando/go-keyring v0.2.6 h1:r7Yc3+H+Ux0+M72zacZoItR3UDxeWfKTcabvkI8ua9s=
 github.com/zalando/go-keyring v0.2.6/go.mod h1:2TCrxYrbUNYfNS/Kgy/LSrkSQzZ5UPVH85RwfczwvcI=
 golang.org/x/sys v0.36.0 h1:KVRy2GtZBrk1cBYA7MKu5bEZFxQk4NIDV6RLVcC8o0k=

internal/cli/cli.go

@@ -43,6 +43,7 @@ type reviewFlags struct {
 	excludes                                                                   []string
 	failBelow                                                                  float64
 	maxDiffBytes                                                               int
+	chunkTokens                                                                int
 }
 
 func reviewCommand() *cobra.Command {
@@ -68,6 +69,7 @@ func reviewCommand() *cobra.Command {
 	command.Flags().Float64Var(&flags.failBelow, "fail-below", 0, "fail when rating is below this value")
 	command.Flags().StringVar(&flags.failSeverity, "fail-on-severity", "", "fail at or above this severity")
 	command.Flags().IntVar(&flags.maxDiffBytes, "max-diff-bytes", 0, "override diff safety limit")
+	command.Flags().IntVar(&flags.chunkTokens, "chunk-tokens", 0, "override diff chunk token limit (default 50000; excludes prompt and tool context)")
 	command.Flags().StringVar(&flags.configPath, "config", "", "explicit repository config path")
 	command.Flags().BoolVar(&flags.noColor, "no-color", false, "disable color output")
 	command.Flags().BoolVar(&flags.verbose, "verbose", false, "enable verbose diagnostics")
@@ -100,6 +102,9 @@ func runReview(command *cobra.Command, flags *reviewFlags) error {
 	if flags.maxDiffBytes != 0 && flags.maxDiffBytes < 10_000 {
 		return dv.New(dv.ExitArguments, "--max-diff-bytes must be at least 10000")
 	}
+	if command.Flags().Changed("chunk-tokens") && flags.chunkTokens < 1000 {
+		return dv.New(dv.ExitArguments, "--chunk-tokens must be at least 1000")
+	}
 	var failBelow *float64
 	if command.Flags().Changed("fail-below") {
 		if flags.failBelow < 1 || flags.failBelow > 5 {
@@ -150,7 +155,7 @@ func runReview(command *cobra.Command, flags *reviewFlags) error {
 		ProviderName: flags.provider, Transport: flags.transport, Model: flags.model, Effort: flags.effort,
 		Base: base, CommittedOnly: flags.committedOnly || flags.pr > 0 || flags.publish,
 		StagedOnly: flags.stagedOnly, Excludes: flags.excludes, ConfigPath: flags.configPath,
-		MaxDiffBytes: flags.maxDiffBytes, FailBelow: failBelow, FailOnSeverity: flags.failSeverity,
+		MaxDiffBytes: flags.maxDiffBytes, ChunkTokens: flags.chunkTokens, FailBelow: failBelow, FailOnSeverity: flags.failSeverity,
 		PublicationRequested: flags.publish, Root: repoRoot, Diagnostics: command.ErrOrStderr(),
 	})
 	if err != nil {

internal/config/config.go

@@ -31,6 +31,7 @@ type Review struct {
 	Exclude      []string       `yaml:"exclude"`
 	MaxDiffBytes int            `yaml:"max_diff_bytes"`
 	ChunkBytes   int            `yaml:"chunk_bytes"`
+	ChunkTokens  int            `yaml:"chunk_tokens"`
 }
 
 type QualityGate struct {
@@ -55,7 +56,7 @@ func Defaults() Repository {
 				"testing": 15, "scope": 10,
 			},
 			MaxDiffBytes: 500_000,
-			ChunkBytes:   180_000,
+			ChunkTokens:  50_000,
 		},
 	}
 }
@@ -107,6 +108,22 @@ func LoadRepository(repo, explicit, trustedRef string) (Repository, error) {
 	if err := decoder.Decode(&value); err != nil {
 		return value, fmt.Errorf("invalid DiffVouch config %s: %w", source, err)
 	}
+	// Preserve explicitly configured byte limits during migration. New configs
+	// use tokens; an explicit choice of both units is ambiguous.
+	var fields struct {
+		Review map[string]yaml.Node `yaml:"review"`
+	}
+	if err := yaml.Unmarshal(raw, &fields); err != nil {
+		return value, err
+	}
+	_, hasBytes := fields.Review["chunk_bytes"]
+	_, hasTokens := fields.Review["chunk_tokens"]
+	if hasBytes && hasTokens {
+		return value, errors.New("configure chunk_tokens or legacy chunk_bytes, not both")
+	}
+	if hasBytes {
+		value.Review.ChunkTokens = 0
+	}
 	if err := validate(value); err != nil {
 		return value, fmt.Errorf("invalid DiffVouch config %s: %w", source, err)
 	}
@@ -120,8 +137,14 @@ func validate(value Repository) error {
 	if value.Provider.DefaultTransport != "" && value.Provider.DefaultTransport != "cli" {
 		return errors.New("repository config cannot select API transport")
 	}
-	if value.Review.MaxDiffBytes < 10_000 || value.Review.ChunkBytes < 10_000 {
-		return errors.New("diff and chunk limits must be at least 10000 bytes")
+	if value.Review.MaxDiffBytes < 10_000 {
+		return errors.New("diff safety limit must be at least 10000 bytes")
+	}
+	if value.Review.ChunkTokens == 0 && value.Review.ChunkBytes < 10_000 {
+		return errors.New("chunk_tokens must be at least 1000, or legacy chunk_bytes at least 10000")
+	}
+	if value.Review.ChunkTokens != 0 && value.Review.ChunkTokens < 1000 {
+		return errors.New("chunk_tokens must be at least 1000")
 	}
 	total := 0
 	for _, name := range []string{"correctness", "security", "maintainability", "testing", "scope"} {