test(ai): scan orchestrator mode literals via AST - #130
Conversation
Keep ADR-0013 and drop the colliding ADR-0005 copies. Record one Unreleased changelog entry, restore the runtime-adapter and post-evaluation transport regressions, and correct the leftover post-chat docstring so it no longer describes a forced route. Co-authored-by: Seongho Bae <[email protected]>
Split auto and verify client lists so a post-chat docstring cannot satisfy the auto policy. Add wire-level verify assertions for citation chat and lineage adjudication, and name both exceptions in the Unreleased changelog. Co-authored-by: Seongho Bae <[email protected]>
A docstring mention of mode="auto" or mode="verify" can no longer satisfy the source-scan contract. Dict-payload clients must contain the JSON literal; post-evaluation keeps the typed default plus forwarded "mode": mode. Co-authored-by: Seongho Bae <[email protected]>
Close the remaining ADR-0013 hole where a docstring JSON fragment could satisfy a whole-file substring scan. The honesty unit now runs the same helpers against a prose-only fixture. Orchestrator vision sends mode=auto; generic OpenAI-compat vision still omits it. Co-authored-by: Seongho Bae <[email protected]>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
This is the landing vehicle for ADR-0013 transport honesty. Keep #117, #106, and draft #109 closed behind it.
What landed. The source scan now walks the AST, so a docstring mode="auto" or a quoted {"mode": "auto"} fragment cannot green the contract. The honesty unit calls those same helpers. Orchestrator-built vision sends mode="auto" on describe(); a generic OpenAI-compatible vision client still omits mode. Wire tests cover every current chat/completions consumer (summary, evaluation, keyman, commitment, entity-relationship, vision, post-chat, adjudication).
Not merge-blocking. The factory and forwarded-auto helpers are still file-global and value-blind (an unused {"mode": "auto"} dict, or mode="auto" plus any x["mode"] = ..., would satisfy the AST scan). Today’s wire tests backstop that. Do not open another mode-scan PR to tighten the helpers; fold that in when the next channel is added.
Before a human approves. Update the data-flow mermaid in ARCHITECTURE.md: the ORC node still says mode=verify for the whole LLM channel, which contradicts the callout this PR added. That diagram is what an operator reads first.
Do not merge yet. Required checks are still queued, and this automation cannot supply the non-author approval. Do not mix related-node captions or analysis-run work into this branch.
Next human action. Wait for Tests / security / required review checks, fix the mermaid line, then approve from a human account. After it lands, the next buyer-facing slice is related-node chip context (— already in flight on #123 / #92), not another transport scan.
Sent by Cursor Automation: Fix Issues
| > not caused by anything in this repo. Ordinary product adapters request | ||
| > `mode="auto"` and are unaffected. Vision built by | ||
| > `orchestrator_vision_client` also sends `mode="auto"`; a generic | ||
| > OpenAI-compatible vision client omits `mode`. |
There was a problem hiding this comment.
The callout is now honest about auto vs verify vs omitted vision mode. The mermaid ORC node above this table still reads mode=verify for the entire LLM channel. Change that node to something like mode=auto (verify for chat/adjudication) before a human approves, so the first diagram an operator sees matches ADR-0013.
|
Superseded by the current architecture. Protected |
2 participants
Summary
Successor to #117. Keep the auto/verify split and wire-level
answer()/judge()assertions, then close the remaining ADR-0013 holes:modeliterals so a docstringmode="auto"or a quoted{"mode": "auto"}fragment cannot satisfy the scan.orchestrator_vision_clientas an ordinary auto consumer: it sendsmode="auto"ondescribe(). GenericOpenAiCompatibleVisionClientstill omitsmodeso OpenAI-compatible gateways do not 400.Prefer this PR over merging #117, #106, or draft #109. Transport-only — do not mix related-node captions.
Product contract
mode="auto"on the wire.mode="verify"on the wire.mode="auto"; a raw OpenAI-compatible vision client omitsmode.Test plan
pytest tests/test_contextual_orchestrator_default_policy.py tests/test_image_content.py— 22 passed, 8 subtestsmode="auto"plus quoted{"mode": "auto"}fails_assert_auto_client/_assert_verify_clientorchestrator_vision_client(...).describe()asserts outboundmode == "auto"OpenAiCompatibleVisionClient.describe()assertsmodeis absentSynthetic fixtures only. No raw LLM API.