Add full-sprint performance and cost benchmark pipeline

[rica-v3]

Summary

• Add a repeatable benchmark sprint-ab pipeline that runs isolated Before/After full-sprint scenarios and emits JSON plus Markdown execution reports.
• Capture provider attempts, duration, prompt size, native token usage, cache usage, tool calls, pricing coverage, workflow outcomes, and quality gates.
• Seed deterministic long request history so the merged prompt-compaction behavior from PR #5 can be measured with compaction disabled and enabled.
• Add bounded execution, durable invocation accounting, provider process registration, process-group cleanup, and fail-closed journal/telemetry reconciliation.
• Document setup, Backfill, Compaction, worked input/output examples, metric semantics, and operational safety in docs/performance_benchmarking.md.

Tracks #2.

Why

PRs #3 through #5 added telemetry, reduced planner checkpoint amplification, and compacted prompt event history. This PR supplies the reproducible integration benchmark needed to quantify those optimizations without treating incomplete or untrusted measurements as valid cost evidence.

Benchmark workflow

TEAMS_RUNTIME_LIVE_BENCHMARK=1 PYTHONPATH=.. \
python -m teams_runtime benchmark sprint-ab \
  --live \
  --runtime-config ../teams_generated/team_runtime.yaml \
  --repetitions 1 \
  --max-invocations 20 \
  --call-timeout-seconds 300 \
  --run-timeout-seconds 1800 \
  --keep-workspaces failures

The Before arm disables prompt compaction. The After arm enables it. Both arms use isolated workspaces and the same deterministic scenario.

Measurement safeguards

• Reserve every provider attempt before launch and reconcile the invocation journal with telemetry identities.
• Gate provider execution until its PID/process group has been durably registered.
• Terminate provider process groups on timeout or worker failure and verify no descendants remain.
• Suppress token coverage and monetary cost when the journal is missing, malformed, unsupported, over budget, or identity reconciliation fails.
• Report native token coverage against all attempted calls, not only calls that emitted telemetry.
• Keep reports privacy-safe by storing counts, digests, and usage metadata rather than prompts or credentials.

First live run

One Before/After pair was executed during implementation:

Metric	Before	After	Delta
Attempted provider calls	8	8	0
Completed / timeout / terminated	5 / 2 / 1	5 / 2 / 1	unchanged
Native token coverage	5/8 (62.5%)	5/8 (62.5%)	unchanged
Wall duration	1,800.030 s	1,800.017 s	-0.013 s
Provider duration	1,659.709 s	1,645.025 s	-14.684 s
Prompt characters	394,728	393,621	-0.28%
Input tokens	10,193,927	21,734,202	+113.21%
Total tokens	10,323,019	21,886,497	+112.02%
Tool calls	132	137	+5
Monetary cost	N/A	N/A	no rate card

Interpretation

The first pair is inconclusive:

• Both arms reached the 30-minute timeout during planning.
• Neither completed a TODO, implementation, QA, commit, or closeout.
• The original fixture did not reach the provider with enough history to trigger compaction.
• The token delta is therefore run-to-run/session variance, not evidence of an optimization regression.
• The scenario has since been corrected so the 50-event Backfill is applied exactly once to the canonical first planning request. A future v2 pair can verify the expected 50-event Before versus 16-included/34-omitted After projection.

Validation

• PYTHONPATH=.. python -m unittest discover -s tests
  • 824 tests passed, 1 skipped
• Focused execution-policy, worker-cleanup, and sprint-benchmark suites passed.
• python -m compileall passed.
• git diff --check passed.
• Independent race/identity review found no high- or medium-severity finding.
• Retained benchmark artifacts passed JSON parsing, credential-pattern, symlink, file-permission, and surviving-process checks.

Follow-up measurement gate

Run the corrected v2 live pair after this benchmark implementation is reviewed. Claim an optimization result only when compaction is observed, workflow outcomes are comparable, invocation identities reconcile, token coverage is disclosed, and pricing coverage is complete when monetary cost is reported.

[rica-v3]

@codex

[chatgpt-codex-connector]

To use Codex here, create a Codex account and connect to github.

[rica-v3]

@codex Please review

[chatgpt-codex-connector]

To use Codex here, create a Codex account and connect to github.

[Copilot]

Pull request overview

This PR introduces a repeatable “sprint A/B” benchmark pipeline to measure full-sprint cost/performance deltas (especially prompt-context compaction), with strong safety controls for live provider execution and privacy-safe reporting.

Changes:

• Add a new benchmark sprint-ab CLI workflow that runs isolated Before/After sprint arms and emits JSON + Markdown reports.
• Introduce bounded provider execution controls (invocation budget, per-call timeout, durable PID/process-group registration, and cleanup) plus prompt-context projection metadata in telemetry.
• Add scenario scaffolding + telemetry reduction/comparison utilities and extensive unit tests covering execution policy, worker cleanup, and telemetry aggregation.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
workflows/sprints/lifecycle.py	Add helper to apply initial-plan confirmation updates to sprint state.
workflows/orchestration/team_service.py	Thread benchmark execution policy + external research toggle through runtime construction; use plan confirmation helper.
runtime/base_runtime.py	Integrate prompt-context projection into invocation telemetry; disable sandbox-bypass retry behavior during benchmarks.
runtime/research_runtime.py	Add allow_external_research policy gate; record prompt-context projection for research decisions.
runtime/model_telemetry.py	Extend telemetry context/aggregation to include prompt projection + tool-call metrics and stricter pricing totals.
runtime/execution_policy.py	Add benchmark execution policy + invocation budget/journaling primitives.
runtime/codex_runner.py	Enforce benchmark sandboxing, environment allowlisting, gated launch, timeouts, journaling, and tool-call parsing.
runtime/benchmark_launcher.py	Add small gatekeeper executable to block provider exec until parent signals readiness.
runtime/internal/intent_parser.py	Pass execution policy through to internal Codex runner.
runtime/internal/goal_sourcing.py	Pass execution policy through to internal Codex runner.
cli.py	Add cmd_benchmark_sprint_ab CLI entrypoint and result rendering.
adapters/cli/commands.py	Wire benchmark sprint-ab subcommand + args into CLI dispatcher.
benchmarking/init.py	Export benchmarking public API surface.
benchmarking/models.py	Define benchmark datamodels, scheduling, and privacy-safe sanitizers.
benchmarking/scenario.py	Create deterministic scenario workspace + backfill history seeding and config hashing.
benchmarking/runner.py	Orchestrate paired runs, retention snapshots, report writing, and safety/coverage gates.
benchmarking/metrics.py	Sanitize telemetry records and compute per-arm metrics + before/after comparisons.
benchmarking/reporting.py	Render the JSON + Markdown benchmark report artifacts.
docs/performance_benchmarking.md	Document benchmark purpose, safety model, setup, and report semantics.
tests/test_model_telemetry.py	Add coverage for tool-call counting, prompt-context projection, and aggregation semantics.
tests/test_execution_policy.py	Add tests for benchmark execution policy, sandboxing, environment filtering, and gated launch behavior.
tests/test_benchmark_worker_cleanup.py	Add tests for worker/provider cleanup, journal reconciliation, and fail-closed behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.