Harden runtime invariants and evidence boundaries

[CoreyLeath-code]

1. About vs code

Claim	Reality	Evidence	Resolution
LLM reasoning/orchestration	No provider call exists; the runtime uses keyword rules	agent/autonomy/planning.py	README and metadata now describe deterministic rule-based planning
Enterprise data tools	Optional modules exist, but default governed runtime wires web search, vectors, and synthesis only	agent/autonomy/runtime.py	Removed this system-level claim; integration evidence is not fabricated
Java planner	Two duplicate Java planners had no JPype/JAR runtime invocation or tests	java/ and Python runtime search	REMOVED
Native vector acceleration	C++ cosine is loaded through ctypes only when a shared library is present; Python is the fallback	agent/agent_core.py	Retained with explicit environment-dependent behavior
Durable checkpoints	Typed run JSON is persisted in SQLite after runtime transitions	agent/autonomy/store.py	Retained and bounded to single-process SQLite semantics

Recommended GitHub About: Deterministic, budgeted Python agent runtime with governed tools, SQLite checkpoints, FastAPI endpoints, and optional C++ cosine-similarity acceleration.

Update the GitHub About description manually; this workflow does not change repository metadata.

2. Planner classification

Deterministic planner. RuleBasedPlanner in agent/autonomy/planning.py proposes keyword-driven typed tasks. A model-backed planner is only a protocol extension point; no LLM orchestration is implemented or claimed.

3. Java decision

REMOVED. The Java sources were duplicate, keyword-driven planners with no runtime integration, JAR invocation, or automated integration test. Removing them is more accurate than preserving an unused language claim.

4. Tests added

Category	Tests	Result
Unit/invariant	terminal execution and unknown-tool persistence	NOT EXECUTED locally
Property	cosine bounds, symmetry, and self-similarity	NOT EXECUTED locally
Fault boundary	unknown planned tool fails closed	NOT EXECUTED locally
Fallback	zero vector and dimension mismatch	NOT EXECUTED locally
Evaluation harness	sample standard deviation and Wilson interval	NOT EXECUTED locally

5. Coverage

Command: pytest tests -v --cov=agent --cov=api --cov=src --cov-report=term-missing
Scope: agent, api, src.
Line/branch coverage: pending CI.
Commit: pending CI head.

No repository-wide or 90% claim is made. The hard coverage threshold remains a follow-up after measuring this broader scope.

6. Mutation testing

NOT EXECUTED. No mutation score is claimed. Add a targeted mutation configuration after the core coverage baseline is measured.

7. Evaluation

No agent-vs-baseline values are introduced. The PR adds tested Wilson and sample-standard-deviation primitives plus an artifact contract for a future deterministic task suite. The existing benchmark remains a local deterministic control-plane microbenchmark.

8. Fault-tolerance findings

The tests establish terminal idempotence, fail-closed unknown-tool handling, pre-execution approval gates, bounded retries/budgets, timeout control return, and checkpoint persistence. They do not establish exactly-once external side effects, distributed execution, or horizontal-concurrency guarantees.

9. Tool results

• Local ruff: NOT EXECUTED (no local checkout/runtime dependencies available).
• Local pytest/coverage: NOT EXECUTED.
• Local mutation testing: NOT EXECUTED.
• Local benchmark: NOT EXECUTED.
• GitHub Actions: pending.

10. Remaining limitations

• No live LLM provider or live-provider evaluation.
• SQLite is a single-process persistence mechanism; no leases or distributed scheduler.
• Checkpoint JSON lacks explicit schema versioning/migration.
• C++ parity is not tested on a native-enabled runner in this PR.
• Evaluation task suite and naive baseline are not yet implemented; no quality result is claimed.
• CI-hosted performance gating remains inappropriate until benchmark variance is characterized.

Summary by CodeRabbit

• Bug Fixes

  • Improved similarity calculations for zero vectors, extreme values, and numerical edge cases.
  • Prevented completed or failed runs from executing again.
  • Tool lookup failures now produce a recorded failed run with an explanatory message.

• Documentation

  • Expanded guidance on runtime behavior, checkpoints, observability, evaluation evidence, and feature limitations.

• Testing

  • Added broader coverage for runtime safeguards, vector similarity, and benchmark statistics.

• Evaluation

  • Added deterministic statistical utilities and documented requirements for future benchmark reporting.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 2634238e-bbfa-4397-8097-7790b5637211

📥 Commits

Reviewing files that changed from the base of the PR and between 20c1740 and 8b0a373.

📒 Files selected for processing (17)

• .github/workflows/ci-cd.yml
• README.md
• agent/agent_core.py
• agent/autonomy/runtime.py
• agent/java/.gitkeep
• benchmarks/eval/README.md
• benchmarks/eval/__init__.py
• benchmarks/eval/statistics.py
• docs/CLAIMS_MATRIX.md
• docs/RUNTIME_INVARIANTS.md
• java/pom.xml
• java/src/com/trojan/ai/Planner.java
• java/src/main/java/com/trojan/ai/Planner.java
• pyproject.toml
• requirements-dev.txt
• tests/test_eval_statistics.py
• tests/test_runtime_invariants.py

---

📝 Walkthrough

Walkthrough

The PR updates runtime terminal and tool-failure handling, stabilizes Python cosine similarity, adds deterministic evaluation statistics and tests, expands coverage configuration, documents evidence boundaries, and removes the Java planner project.

Changes

Runtime and vector invariants

Layer / File(s)	Summary
Runtime execution and cosine similarity agent/agent_core.py, agent/autonomy/runtime.py, tests/test_runtime_invariants.py, requirements-dev.txt	Cosine similarity now handles extreme values and zero vectors. Terminal runs are not re-executed. Missing tools persist failed run state. Tests add property-based vector checks and runtime failure coverage.
Evaluation statistics benchmarks/eval/*, tests/test_eval_statistics.py	The evaluation package adds sample mean and standard deviation calculation plus Wilson confidence intervals. Tests cover valid results and invalid inputs.

Repository scope and evidence

Layer / File(s)	Summary
Coverage and project metadata .github/workflows/ci-cd.yml, pyproject.toml	Coverage collection now includes agent. The project description identifies the deterministic runtime and checkpoint model.
Documentation and planner scope README.md, docs/CLAIMS_MATRIX.md, docs/RUNTIME_INVARIANTS.md, benchmarks/eval/README.md, java/pom.xml, java/src/.../Planner.java, java/src/main/.../Planner.java, agent/java/.gitkeep	Documentation narrows planner claims, records evidence boundaries and runtime limitations, and defines benchmark reporting requirements. The Java planner project and placeholder comments are removed.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

• CoreyLeath-code/HelixAgent#4: Updates the same CI coverage configuration for the agent package.
• CoreyLeath-code/HelixAgent#27: Introduced functionality refined here, including cosine_sim and AutonomousRuntime.run.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/evidence-runtime-hardening

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}