Skip to content

Fix Orchestrate planning contracts and native document handoffs - #1904

Merged
Paul Lizer (paullizer) merged 2 commits into
paullizer-react-v2-uifrom
paullizer-orchestrate-planning-failures
Nov 7, 2026
Merged

Paul Lizer (paullizer) merged 2 commits into
paullizer-react-v2-uifrom
paullizer-orchestrate-planning-failures

Conversation

@paullizer

@paullizer Paul Lizer (paullizer) commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Similar Orchestrate requests could fail before execution because advertised Analyze outputs conflicted with server defaults, while native document actions could not declare selected-source use. Align effective output variants and provide precise contract-aware recovery within the existing single correction allowance.
  • Enforce exact workspace-document bindings for registered Document Search actions, rechecking authority and source revisions before tool calls. Preserve those bindings across the executor handoff, mixed workspace/action retention, citation projection, and final composition without adding redundant Search/Analyze work.
  • Add content-free, debug-gated checkpoints in console and Application Insights using hashed identities, counts, known operations, and safe codes. Preserve planning phase/rule/attempt information through streaming, saved questions, reload, and explicit retry.
  • Add authenticated end-to-end Auto-run coverage for PDF/Word, successful saved cited answers, denied reads, and debug on/off. Update related documentation and the integrated application version to 0.261.420 (original implementation: 0.261.419).

Retained native sources keep their existing provenance and conversation/run access semantics; current integration access checks remain in place. Native metadata is not a conformance guarantee. No new route, setting, extraction engine, browser runtime asset, deployment, or live conversation change is introduced by this fix.

The original generic production failures did not retain their rejected model JSON. This PR fixes reproduced shared contract/integration gaps without claiming an exact historical field-level cause for every failed turn.

The latest React V2 base is integrated in merge commit c8a1b950c. The logging conflict was resolved by preserving both the native orchestration diagnostics and the base's saved-findings correlation fields; the base's Sources inspector behavior is unchanged by the resolution.

Linked issue

N/A. This work was intentionally tracked in the session without creating an issue.

Release Notes & Latest Features

  • New Feature
  • Bug Fix
  • UI Enhancement
  • Breaking Change
  • Internal only

Is this visible to end users?

  • Yes
  • No

Is this admin-facing (Admin Settings, governance, deployment, config)?

  • Yes
  • No

Uses the existing debug-logging switch; no new admin setting or deployment change.

Should this become a Latest Feature card?

  • Yes
  • No
  • Already added

Screenshot needed for the card?

  • Yes
  • No
  • Attached

Version bump

  • application/single_app/config.py VERSION third segment bumped, or not needed because this is docs-only
  • deployers/version.txt bumped, or not needed because deployers/ was not changed

Integrated application version: 0.261.419 -> 0.261.420. The original implementation bumped 0.261.418 -> 0.261.419; current-version headers now reflect the integrated version while implementation-version metadata remains historical. No deployer files changed.

Testing / validation

  • Post-integration browser selection: 75 passed, covering native bound-source review and classic/V2 saved-findings Sources inspection. The merged production TypeScript/Vite build passed into isolated test artifacts. Documentation coverage passed 7/7, site quality passed 6/6, and the committed PR diff passed whitespace checks.
  • Original implementation's related orchestration regression selection: 507 passed, 184 subtests passed. Includes real planner/compiler, native action filters, authenticated Auto-run publication, source selection, retained lineage/citations, completeness, and saved-turn recovery with external I/O isolated.
  • Original implementation's reviewed SDK storage gate: 2,015 passed, 431 subtests passed, with network blocked before collection. The changed-owner/double selection also passed 251 tests independently.
  • Original implementation's browser source-review and planning-retry selection: 16 passed. Route policies and logger initialization: 31 passed. Production V2 TypeScript/Vite build and whitespace checks passed. Documentation coverage: 7/7; site quality: 6/6.

Counts are separate, overlapping selections, not a combined total. The full original gates are recorded historical results, not a claim that all were rerun for this merge. Broader unrelated baseline failures were reproduced against pre-fix production code and are documented in the fix explanation; they are not suppressed or counted as passing. Validation does not claim live-model accuracy or a deployed success rate.

Exact commands and results

Python integration commands used the application and functional-test directories on PYTHONPATH.

Post-integration production V2 build - passed; generated output isolated in test artifacts:

npm --prefix .\application\v2_ui run build -- --outDir ..\..\ui_tests\artifacts\saved-analysis

Post-integration browser selection - passed (75 tests):

$env:PYTHONDONTWRITEBYTECODE = '1'
python -m pytest -q -p no:cacheprovider --disable-warnings --tb=short --show-capture=no .\ui_tests\test_v2_orchestration_actions.py::test_native_action_bound_sources_are_visible_and_can_be_narrowed .\ui_tests\test_chat_saved_analysis.py

The saved-analysis fixture initially lacked its documented production CSS build artifact. Building it with the documented command above resolved setup; no test was changed or skipped to bypass that failure.

Post-integration documentation - coverage passed 7/7 and quality passed 6/6:

python .\functional_tests\test_docs_app_surface_coverage.py
python .\functional_tests\test_docs_site_quality.py

Post-integration committed PR diff - passed:

git --no-pager diff origin/paullizer-react-v2-ui...HEAD --check

Original related regression selection - passed (507 tests, 184 subtests):

python -m pytest -q -p no:cacheprovider --disable-warnings --tb=short --show-capture=no .\functional_tests\test_orchestration_planner_contract_recovery.py .\functional_tests\test_orchestration_document_planning_recovery.py .\functional_tests\test_orchestration_document_action_binding.py .\functional_tests\test_orchestration_document_action_lineage.py .\functional_tests\test_orchestration_document_action_end_to_end.py .\functional_tests\test_orchestration_planning_persistence.py .\functional_tests\test_orchestration_source_kind_planning.py .\functional_tests\test_orchestration_selected_source_planning.py .\functional_tests\test_orchestration_action_planning.py .\functional_tests\test_orchestration_action_runtime.py .\functional_tests\test_orchestration_action_catalog.py .\functional_tests\test_orchestration_result_contracts.py .\functional_tests\test_orchestration_registry_contract.py .\functional_tests\test_orchestration_external_sources.py .\functional_tests\test_orchestration_followup_citation_provenance.py .\functional_tests\test_orchestration_available_information.py .\functional_tests\test_document_analysis_observation_identity.py .\functional_tests\test_document_source_scope_identity.py

Original full reviewed SDK gate - passed (2,015 tests, 431 subtests):

$env:PYTEST_ADDOPTS = '-vv -o faulthandler_timeout=90'
python .\scripts\check_cosmos_response_boundaries.py --run-contracts

Inventory review - passed (1,094 owners, 425 doubles, zero review gaps):

python .\scripts\check_cosmos_response_boundaries.py

Original browser selection - passed (16 tests):

python -m pytest -q -p no:cacheprovider --disable-warnings --tb=short .\ui_tests\test_v2_orchestration_actions.py::test_native_action_bound_sources_are_visible_and_can_be_narrowed .\ui_tests\test_v2_orchestration_planning_retry.py

Original route policies and logger initialization - passed (31 tests):

python -m pytest -q -p no:cacheprovider --disable-warnings --tb=short .\functional_tests\route_tests\test_route_blueprint_policy_inventory.py .\functional_tests\route_tests\test_route_unauthenticated_policy_contract.py .\functional_tests\route_tests\test_route_policy_test_coverage.py .\functional_tests\test_appinsights_logging_recursion_guard.py

Original production V2 build - passed; generated output isolated in test artifacts:

npm --prefix .\application\v2_ui run build -- --outDir ..\..\ui_tests\artifacts\orchestration-plan-editor

Original documentation - inventory regenerated; coverage passed 7/7 and quality passed 6/6:

python .\scripts\build_docs_inventory.py
python .\functional_tests\test_docs_app_surface_coverage.py
python .\functional_tests\test_docs_site_quality.py

The regenerated application-surface inventory matched the committed content; no surface item was added or renamed.

Original whitespace - passed before committing:

git -c core.whitespace=blank-at-eol,blank-at-eof,space-before-tab,cr-at-eol diff --check

Documentation

  • Release notes updated, or not needed
  • Feature documentation updated, or not needed
  • Fix documentation updated, or not needed

Release notes were left unchanged pending review. Updated action access, Document Search, planning recovery, and logging references. Added docs/explanation/fixes/ORCHESTRATION_PLANNER_CONTRACT_RECOVERY_FIX.md with root causes, behavior, full-path validation, privacy boundaries, and baseline limitations; its integrated-version note records the merged logging contracts and validation.

Security checklist

  • New Flask routes include @swagger_route(security=get_auth_security())
  • Settings sent to non-admin frontends use sanitize_settings_for_user()
  • Browser JavaScript is served from local SimpleChat static assets only; no CDN-hosted JS
  • No secrets, keys, connection strings, or local-only artifacts are included

No new routes or settings-rendering paths were added. Existing route policies, action restrictions, current source authority, conditional SDK writes, and exact-data completeness checks remain enforced. Debug checkpoints exclude raw source/model/provider payloads; session investigation artifacts and generated browser-test assets are not included.

Align planner defaults and output contracts, enforce native document action bindings through retention and citations, and add content-free debug checkpoints and authenticated end-to-end coverage. Bump the application version to 0.261.419.

Co-authored-by: Copilot App <[email protected]>
Retain both native document-action diagnostics and saved-findings telemetry when resolving the logging allowlist conflict. Advance the integrated application version to 0.261.420 and synchronize current-version headers.

Co-authored-by: Copilot App <[email protected]>
@paullizer
Paul Lizer (paullizer) merged commit a974d36 into paullizer-react-v2-ui Nov 7, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant