test(ai): require payload literals for orchestrator mode scans - #117
test(ai): require payload literals for orchestrator mode scans#117cursor[bot] wants to merge 3 commits into
Conversation
Keep ADR-0013 and drop the colliding ADR-0005 copies. Record one Unreleased changelog entry, restore the runtime-adapter and post-evaluation transport regressions, and correct the leftover post-chat docstring so it no longer describes a forced route. Co-authored-by: Seongho Bae <[email protected]>
Split auto and verify client lists so a post-chat docstring cannot satisfy the auto policy. Add wire-level verify assertions for citation chat and lineage adjudication, and name both exceptions in the Unreleased changelog. Co-authored-by: Seongho Bae <[email protected]>
A docstring mention of mode="auto" or mode="verify" can no longer satisfy the source-scan contract. Dict-payload clients must contain the JSON literal; post-evaluation keeps the typed default plus forwarded "mode": mode. Co-authored-by: Seongho Bae <[email protected]>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
ADR-0013 payload literals are a real lock over #106: live adapters send auto/verify, and wire tests call answer() / judge(). Do not merge this head as the final transport contract.
Residuals on 32b6d8a:
test_docstring_mode_mention_is_not_a_payload_literalonly checks thatmode="auto"is not the substring"mode": "auto". It never runs the scan against a file. A later edit that ORs the docstring spelling back in would still pass.- Whole-file
assertIn("mode": "auto")still greens if a class docstring quotes that JSON after the real payload is deleted. Same class of hole as #106, one spelling later. lineageweave/image_content.pyis an orchestrator chat consumer (orchestrator_vision_client) and is not inAUTO_CLIENTS. A generic OpenAI-compatible vision client must keep omittingmodeso unknown-field gateways do not 400.
Successor cursor/bc-c1bda94f-3013-4808-8be0-a9bc8c075d80-c501 (b1f1db6) walks the AST, runs those same helpers against a prose-only fixture (including quoted JSON), and sends mode="auto" from orchestrator_vision_client. Prefer that PR over this one, and over #106 / draft #109. Keep this lock transport-only — do not mix related-node captions.
Focused checks on the successor: pytest tests/test_contextual_orchestrator_default_policy.py tests/test_image_content.py — 22 passed, 8 subtests. CodeRabbit CLI 0.7.3 is present but auth login --agent stays on awaiting_browser_auth here, so this is not a CodeRabbit CLI report.
Sent by Cursor Automation: Fix Issues
| import unittest | ||
|
|
||
| ROOT = Path(__file__).resolve().parents[1] | ||
| AUTO_CLIENTS = ( |
There was a problem hiding this comment.
AUTO_CLIENTS omits lineageweave/image_content.py. orchestrator_vision_client POSTs /v1/chat/completions and is an ordinary ADR-0013 consumer. Add it as a factory-auto case (mode="auto" on the factory, written onto the body) rather than stuffing mode into generic OpenAiCompatibleVisionClient — that path must keep omitting mode so OpenAI-compatible gateways do not 400.
| self.assertIn(_TYPED_AUTO_DEFAULT, source) | ||
| continue | ||
| self.assertIn( | ||
| _PAYLOAD_AUTO, |
There was a problem hiding this comment.
Whole-file assertIn(_PAYLOAD_AUTO, source) still passes if the only remaining "mode": "auto" lives inside a class docstring. Walk the AST for dict literals (and the post-evaluation "mode": mode / typed-default exception) so prose JSON cannot satisfy ADR-0013.
| f"{relative} must send verify, not a payload-level auto default", | ||
| ) | ||
|
|
||
| def test_docstring_mode_mention_is_not_a_payload_literal(self) -> None: |
There was a problem hiding this comment.
This unit asserts that mode="auto" is not the substring "mode": "auto". It never calls _source(), never points AUTO_CLIENTS at a fixture, and never expects the auto/verify scan to fail. Point the same assertion helper used on production clients at a prose-only file (docstring mode="auto" plus quoted {"mode": "auto"}) and expect AssertionError.
|
Superseded by protected |
2 participants
Summary
answer()/judge()assertions."mode": "auto"/"mode": "verify"literals so a class docstring cannot satisfy ADR-0013."mode": modeplusmode: str = "auto").mode="auto"mention is not a payload literal.This branch is #106 (
a815440) plus the payload-literal lock. Prefer this PR over merging #106 or draft #109.Product contract
mode="auto"on the wire.mode="verify"on the wire.Test plan
pytest tests/test_contextual_orchestrator_default_policy.py— 3 passed, 7 subtestspost_summary.py's payload"mode": "auto"with"mode": "verify"fails the auto scan while the class docstring still saysmode="auto"Synthetic fixtures only. No raw LLM API.