Skip to content

Fix orchestration review contracts and context-only recovery - #1901

Merged
Paul Lizer (paullizer) merged 2 commits into
microsoft:paullizer-react-v2-uifrom
paullizer:paullizer-orchestration-review-failure
Nov 6, 2026
Merged

Paul Lizer (paullizer) merged 2 commits into
microsoft:paullizer-react-v2-uifrom
paullizer:paullizer-orchestration-review-failure

Conversation

@paullizer

@paullizer Paul Lizer (paullizer) commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Fix prepared answers being withheld when the extra reviewer changed application-owned checklist IDs (completeness_requirements_changed). Known obligations now use a closed keyed review contract with bounded provider-format compatibility and precise repair feedback.
  • Give review and correction the complete authorized original message as source content. Eligible single-step, context-only prose in Auto can retain its draft after a checker error; real content findings, gathered/substantive work, explicit On, and required validation/access/safety gates remain blocking.
  • Show the actual completed review basis or format-only validation in plan details. Skipped-review diagnostics stay out of the chat, never appear as a passed review, and retain source-fingerprint, masking, ownership, cancellation and correction-budget safeguards.
  • Update regression coverage, related documentation and reviewed Cosmos module fingerprints, and bump the application to 0.261.418. Preserve the current base's task/phase progress and native-citation behavior. No normal-chat, deployer, route or dependency changes in the PR diff; no production deployment or conversation replay.

Linked issue

N/A - no associated issue.

Release Notes & Latest Features

  • New Feature
  • Bug Fix
  • UI Enhancement
  • Breaking Change
  • Internal only

Is this visible to end users?

  • Yes
  • No

Is this admin-facing (Admin Settings, governance, deployment, config)?

  • Yes
  • No

Should this become a Latest Feature card?

  • Yes
  • No
  • Already added

Screenshot needed for the card?

  • Yes
  • No
  • Attached

Existing administrative completeness settings remain the outer bound; no new setting or card is introduced. Release notes are unchanged in this PR.

Version bump

  • application/single_app/config.py VERSION third segment bumped, or not needed because this is docs-only
  • deployers/version.txt bumped, or not needed because deployers/ was not changed

Application: current base 0.261.417 -> 0.261.418. Deployer version unchanged because no deployer files changed.

Testing / validation

  • After synchronizing base a78255ac9: 196 functional tests and 51 subtests passed; 111 Python Playwright tests passed. This retains both the base's confirmed task/phase headline and this fix's plan-only review diagnostics.
  • Typecheck, production/component-test builds, documentation coverage/quality and the zero-gap SDK inventory check passed again after the base merge.
  • Original pre-base validation: 165 targeted backend tests and 51 subtests; 93 UI tests; 2,005 offline Cosmos tests and 431 subtests; 22 assets/model tests and 64 subtests passed. The full offline SDK matrix was not repeated for the base synchronization.
  • Desktop light/mobile dark, stale-candidate handling, quiet chat recovery and measured text contrast of at least 4.5:1 are covered. Browser-sink checks, deterministic UI checks and diff checks passed.
Validation commands and results

After base sync: passed 196 functional tests and 51 subtests

python -m pytest -q --tb=short --disable-warnings -p no:cacheprovider .\functional_tests\test_task_completeness_engine.py .\functional_tests\test_task_completeness_runtime.py .\functional_tests\test_orchestration_available_information.py .\functional_tests\test_orchestration_plan_schema.py .\functional_tests\test_orchestration_invoke_prompt_contract.py .\functional_tests\test_orchestration_model_selection.py .\functional_tests\test_orchestration_terminal_wait_retry.py .\functional_tests\test_orchestration_native_citation_delivery.py .\functional_tests\test_orchestration_attachment_native_evidence.py

After base sync: passed 111 UI tests

python -m pytest -q --tb=short --disable-warnings -p no:cacheprovider .\ui_tests\test_v2_orchestration_quiet_activity.py .\ui_tests\test_task_completeness_ui.py .\ui_tests\test_v2_orchestration_recovery.py .\ui_tests\test_v2_stream_citation_delivery.py

Original pre-base matrix: passed 2,005 tests and 431 subtests, with tokenizer caches warmed before network blocking

python .\scripts\check_cosmos_response_boundaries.py --run-contracts

Original assets/model selection: passed 22 tests and 64 subtests

python -m pytest -q --tb=short --disable-warnings -p no:cacheprovider .\functional_tests\test_v2_ui_local_assets.py .\functional_tests\test_model_reasoning_capability_resolution.py

Passed again after base sync: typecheck, production build and component-test build

npm --prefix .\application\v2_ui run typecheck -- --pretty false
npm --prefix .\application\v2_ui run build
npm --prefix .\application\v2_ui run build -- --outDir ..\..\ui_tests\artifacts\orchestration-plan-editor

Passed again after base sync: SDK inventory check, inventory regeneration, 7/7 coverage checks and 6/6 site-quality checks

python .\scripts\check_cosmos_response_boundaries.py
python .\scripts\build_docs_inventory.py
python .\functional_tests\test_docs_app_surface_coverage.py
python .\functional_tests\test_docs_site_quality.py

Passed: whitespace/diff check

git -c core.whitespace=blank-at-eol,blank-at-eof,space-before-tab,cr-at-eol diff --check

Known baseline limitation: an auxiliary combined assets/GPT-6-parameter/capabilities selection returned 2 pre-existing failures and 30 passes. test_gpt6_reasoning_model_parameters.py extracts ModelEndpointBehavior into an isolated namespace without model_uses_mai_chat_api, failing its two non-reasoning-model cases. The production class AST was unchanged from the original base revision, and that unrelated fixture was left unchanged. The real-module selection above passes. The failing selection was:

python -m pytest -q --tb=short --disable-warnings -p no:cacheprovider .\functional_tests\test_v2_ui_local_assets.py .\functional_tests\test_gpt6_reasoning_model_parameters.py .\functional_tests\test_model_reasoning_capability_resolution.py

Scripted provider replies establish control flow and contract behavior, not an empirical semantic-accuracy guarantee. Production was not deployed or replayed.

Documentation

  • Release notes updated, or not needed
  • Feature documentation updated, or not needed
  • Fix documentation updated, or not needed

Updated the completeness feature, orchestration admin guide, plan-editing guide and chat-control reference. Added docs/explanation/fixes/ORCHESTRATION_CONTEXT_ONLY_REVIEW_RECOVERY_FIX.md with root cause, compatibility, safeguards and validation evidence. Version references for this fix now use 0.261.418 while preserving the base's 0.261.417 feature history. Release notes were offered but not changed without confirmation.

Security checklist

  • New Flask routes include @swagger_route(security=get_auth_security())
  • Settings sent to non-admin frontends use sanitize_settings_for_user()
  • Browser JavaScript is served from local SimpleChat static assets only; no CDN-hosted JS
  • No secrets, keys, connection strings, or local-only artifacts are included

No routes or browser settings payloads changed in this PR relative to its base. Required source/access/safety checks remain mandatory, and skipped optional review is never represented as a successful semantic assessment.

Stabilize application-owned review obligations, ground checking in the original message, and add narrowly scoped Auto recovery for checker errors. Preserve strict content, source-access and required-output gates, show truthful plan validation outcomes, and update coverage and documentation for version 0.261.417.

Co-authored-by: Copilot App <[email protected]>
Resolve the shared UI-test version header, retain upstream task/phase progress and native-citation changes, and align review recovery with application version 0.261.418. Verified 196 functional tests and 111 UI tests plus typecheck, builds, documentation and SDK inventory checks.

Co-authored-by: Copilot App <[email protected]>
@paullizer
Paul Lizer (paullizer) merged commit 34df840 into microsoft:paullizer-react-v2-ui Nov 6, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant