fix(strix): retry exact model-quality warnings

[seonghobae]

Summary

• Treat Strix's exact MODEL QUALITY WARNING / not a recommended frontier model output as retryable model evidence.
• Continue through the configured fallback sequence; do not neutralize warning-class failures.
• Keep MEDIUM+ findings, generic warnings, timeouts, and provider failures fail-closed.
• Add regression coverage and APA 7 doctoring.

Root cause

The public-repository NVIDIA NIM scan for naruon#1432 produced one LOW finding and Strix's exact model-quality warning. The gate correctly marked the warning as non-clean evidence but did not classify it as retryable, so it stopped before trying the configured fallback models.

Verification

Exact-head hosted evidence for 843ee9aa6d54053e4d32e700e71ccc120e102cdd against protected main@55a8b576725451dfe0a21a57d36a2f1a41619b24:

• Strix Changed Path Quality CI run 32423950648, job 96601801336: exact SHA checkout; 1,284 central tests and 16 subtests passed.
• Exact Artifact SBOM Attestation Quality run 32423950561, job 96601800969: exact SHA checkout; 49 contracts passed; 226 statements / 82 branches at 100%.
• Organization Commercial Readiness Loop Quality CI run 32423950646, job 96601801401: exact SHA checkout; 34 tests; 380 statements / 88 branches at 100%.
• The current head differs from previously reviewed a6ab3074490409a2c1677af3650bc0428d9a5fc6 only by the non-destructive protected-main integration; the four PR-owned paths are unchanged.
• Security/SBOM checks and an exact-current-head formal verdict remain required; queued evidence is non-passing.

References

See docs/doctoring/strix-quality-warning-fallback.md for APA 7 references and rollback guidance.

---

[coderabbitai]

Warning

Review limit reached

@seonghobae, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 18 minutes

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

Wait for the limit to reset, then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 2e103ac9-b599-4248-90c2-41c86271d4f8

📥 Commits

Reviewing files that changed from the base of the PR and between fce028b and f6c2404.

📒 Files selected for processing (7)

• CHANGELOG.md
• docs/doctoring/strix-quality-warning-fallback.md
• requirements-pip-audit-ci-hashes.txt
• scripts/ci/organization_commercial_readiness_loop.py
• scripts/ci/strix_quick_gate.sh
• scripts/ci/test_strix_quick_gate.sh
• tests/test_strix_nvidia_nim_not_found_fallback.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.

[seonghobae]

Exact commit a6ab307 was pushed and auto-merge is enabled. Targeted checks remain green: 12 Python regression tests, 14 Python workflow-contract tests with 16 subtests, Bash syntax, actionlint, and diff hygiene.

The broad local shell harness was stopped after its sparse checkout reported absent CodeGraph/frontend fixture paths; those are local materialization gaps, not failures in the changed Strix classifier. The protected CI run is the authoritative full-repository verification.

[seonghobae]

Current-head validation for a6ab307: the narrow Strix model-quality warning classifier now marks only the exact provider warning as retryable, preserves generic warning/fatal/timeout failures, and routes the primary attempt to configured fallback models. Passed: 12 focused fallback tests, bash -n, and git diff --check. Shellcheck reports only pre-existing SC2034/SC2329 warnings outside this change. Please review this exact head; merge remains gated on an independent non-author approval and terminal protected checks.

[seonghobae]

@opencode-agent Please review the current PR head a6ab307. Verify the exact model-quality warning classifier, fallback routing, current mergeability, changed-file scope, and required checks. Do not approve a stale head.

[seonghobae]

Current-head review for a6ab3074490409a2c1677af3650bc0428d9a5fc6 against protected main@6479989bbff475404cc2cccc468d5fb1d6c632e5.

The shared failure reproduced on central PRs #1085, #1078, #1065, #930, and #928 was a stale Strix fallback contract: the outer workflow tests did not recognize the vulnerability-count signal while the retry classifier lacked the exact model-quality warning. This PR repairs that common root without neutralizing reported vulnerabilities. Focused proof: 12 Strix fallback tests passed, Bash syntax passed, Interrogate 100%, and git diff --check passed. The broad quick-gate harness remains an environment-sensitive long-running check; protected CI is authoritative. No further actionable finding was found. This is comment-only; independent protected review and terminal Checks remain required.

[seonghobae]

@opencode-agent review @cwl-noema-review review

Review exact current HEAD a6ab307 against protected main. Confirm the narrow Strix model-quality retry classification and that reported vulnerabilities remain blocking.

[seonghobae]

@opencode-agent review exact current HEAD a6ab307. Inspect the complete diff, validate security and regression behavior, and publish only evidence bound to this SHA. Do not transfer predecessor approval, modify the branch, or merge.

[seonghobae]

The failed Strix evidence on #1088 run 32401768402 confirms the exact warning covered here: MODEL QUALITY WARNING and is not a recommended frontier model. Review current HEAD a6ab3074490409a2c1677af3650bc0428d9a5fc6 against that production trace and re-run the pending scan-pr-queue/Strix Checks.

[seonghobae]

@opencode-agent review\nHead SHA: a6ab307\nRe-run the current-head required workflow set and report the terminal result. Prior scan-pr-queue/Strix cancellation or failure must be distinguished from the latest exact-head run.

[seonghobae]

Exact-head formal review requested for 843ee9aa6d54053e4d32e700e71ccc120e102cdd. Hosted evidence now binds the actual checkout to run 32423950648 (1,284 tests + 16 subtests), run 32423950561 (49 contracts, 226/82 100%), and run 32423950646 (34 tests, 380/88 100%). This request is review-only; queued required checks remain non-passing and no merge/branch mutation is authorized.

[seonghobae]

Current-head verification: live head 843ee9aa6d54053e4d32e700e71ccc120e102cdd is cleanly based on protected main@55a8b576725451dfe0a21a57d36a2f1a41619b24. The focused regression suite passes (12 passed), Bash syntax and diff checks pass, and exact-head Strix/security/SBOM checks are successful; coverage-evidence remains queued and therefore non-passing. Please obtain an independent approval against this exact head. Merge remains gated on terminal protected Checks and that approval.

[seonghobae]

Reviewed exact HEAD a6ab3074490409a2c1677af3650bc0428d9a5fc6. The Strix classifier now treats the exact model-quality warning as retryable infrastructure evidence, preserves fail-closed handling for real findings/provider failures, and has a direct fallback regression. Focused tests: 12 passed; Ruff, compileall, interrogate 100%, Bash syntax, and diff checks passed. Enabling normal squash auto-merge; hosted exact-head Strix/Security checks and independent review remain authoritative.

[seonghobae]

Exact-head review disposition

• Exact head: bfb18549
• Base used for the current branch integration: 731af58e954901c4f1cc853231c592abb1eaf617
• Source finding fixed: is_model_quality_warning now requires both the explicit MODEL QUALITY WARNING marker and a provider-qualified recommended-frontier-model detail. Ordinary target output and detail-only output do not trigger fallback.
• Regression evidence: 170 focused Python tests pass, including the positive/header-only/detail-only classifier cases; bash -n scripts/ci/strix_quick_gate.sh scripts/ci/test_strix_quick_gate.sh and git diff --check pass.
• Accepted informational findings: existing retryable provider-warning coverage is additive; vulnerability reports remain fail-closed before fallback; no production behavior is weakened.
• Rollback: revert the exact source commit 4d397eab (or the merge-result commit containing it) if post-merge canary evidence shows a source regression.
• Decision: normal protected merge only after an independent approval and all required exact-head checks are terminal-success. No bypass or stale evidence is used.

[seonghobae]

Exact-head follow-up disposition

• Exact head: 07fa2a69
• Removed the redundant is_model_quality_warning branch from infrastructure detection; the existing token-bounded warning detector already marks the explicit warning as infrastructure evidence. The provider-qualified classifier remains in is_model_retryable_error, where its behavior is material.
• Kept the provider-qualified/header-bound quality-warning regex deliberately strict. A future Strix wording/provider-format change must add a current exact-log fixture before widening it; fail-closed non-retry is safer than accepting target text as provider evidence.
• Verification: 170 focused tests pass; bash -n and git diff --check pass.
• Decision: normal protected merge only after an independent approval and all required exact-head checks are terminal-success. No bypass or stale evidence is used.

[seonghobae]

Exact-head maintainer review (2026-08-21): checked the 4-file diff, including the production is_model_quality_warning / is_model_retryable_error path and fallback failure semantics. The warning remains an infrastructure/failure signal; a fallback must still produce clean evidence, and MEDIUM+ findings remain blocking. Devin comments were dispositioned as informational; no unresolved source finding was identified. Local exact-head checks: python -m unittest tests.test_strix_nvidia_nim_not_found_fallback = 12 passed; bash scripts/ci/test_strix_quick_gate.sh completed with expected NO_CONCLUSION fixture assertions. Hosted state remains 16 nonterminal Checks, 0 failed, and no formal exact-head APPROVED review. Decision: WAIT_AND_REMEDIATE; no merge or bypass.

[seonghobae]

@OpenCode Please perform an independent substantive review of exact head 07fa2a6. Read the complete 4-file diff, report changed-file walkthrough, security/behavior boundary, test gaps, residual uncertainty, and explicit verdict. Do not review an older head.

[seonghobae]

Exact-head verification: 07fa2a6 against base 731af58. Strix fallback contract tests: 12 passed; bash -n and git diff --check passed. Current review comments are informational: vulnerability findings remain fail-closed, the wording limitation is documented, and the redundant branch was removed at this head. Hosted Checks are queued with no current failure; protected Checks and independent approval remain required. Normal path only; no bypass.

[seonghobae]

Exact-head review disposition.

• Head: 07fa2a69c2820d7eff348fcacc017154e85748f6
• Base: 731af58e954901c4f1cc853231c592abb1eaf617
• Scope: retry only a structured Strix model-quality warning through the existing configured cross-model fallback; application output is not treated as provider evidence.
• Review threads: none unresolved.
• Local verification: Strix fallback/path/hash contracts — 23 passed plus 16 subtests; Ruff — passed; bash -n scripts/ci/strix_quick_gate.sh — passed; git diff --check — passed.
• Normal squash auto-merge is already enabled. Required Checks and independent approval remain authoritative; no bypass or stale evidence.

[seonghobae]

Exact-head review and verification

• Repository: ContextualWisdomLab/.github
• Pull request: fix(strix): retry exact model-quality warnings #1177
• Exact head SHA: ad0e96068a9a81b150870dadfa61865bf525c076
• Exact base SHA: dd58a88391e44a32fb399f7407f508d8e73cc1c7
• Scope: Strix retries the exact NVIDIA model-quality warning before provider fallback; vulnerability and incomplete-scan paths remain fail-closed.
• Review disposition: all current review threads are resolved. Informational observations about classifier overlap, conservative fallback, and wording coupling were checked against the current shell path and retained as non-blocking.
• Local verification: focused Strix tests 12 passed; full suite 1292 passed, 16 subtests passed; statement and branch coverage 100% (8127/8127, 3144/3144); bash -n scripts/ci/strix_quick_gate.sh; Ruff; compileall; and git diff --check passed.
• Hosted verification: prior exact-head runs were cancelled before any job steps by shared queue/concurrency control. All cancelled exact-head runs were rerun; no cancelled result is being reused.
• Decision: NORMAL_MERGE path with existing squash auto-merge enabled; no ruleset bypass, force push, direct protected-branch push, or approval fabrication.

[seonghobae]

Current exact head: b2cdb96a86a116533605fa6180abd7873acd524a, base 6a79f075d62b5f8dcfef6145960f1c42c147f273.

The exact-head policy failure was a real contract-test defect: the PR's scheduler source had already adopted the current repository-scoped dispatch fallback and metadata-free workflow_run cancellation, while scripts/ci/test_strix_quick_gate.sh still asserted the predecessor github.run_id and shorter cancellation expression. Updated both assertions to the live source contract.

Local verification: bash -n scripts/ci/test_strix_quick_gate.sh, git diff --check, and python3 -m pytest -q tests/test_strix_nvidia_nim_not_found_fallback.py (12 passed). Hosted Checks are invalidated and must rerun on this head. Decision: WAIT_AND_REMEDIATE until all required Checks and an independent exact-head approval are present.

[seonghobae]

Current-head audit

• Exact head: b2cdb96a86a116533605fa6180abd7873acd524a
• Exact base: 6a79f075d62b5f8dcfef6145960f1c42c147f273
• Review threads: 0 unresolved; no qualifying approval is currently recorded.
• Local exact-head verification: targeted model-quality fallback tests 12 passed; shell syntax, compileall, and diff check passed. The larger shell harness was previously bounded and terminated after exceeding its local timeout, so it is not claimed as passed.
• Hosted checks have no current source failure; the visible failure is inherited pip-audit pip 26.1.2 / PYSEC-2026-3721, owned by dependency repair PR fix(security): repair pip audit and schedule orchestrator review #1198. Strix remains in progress.
• Decision: WAIT_AND_REMEDIATE; normal auto-merge remains enabled; no force merge.

[seonghobae]

Exact-head audit update\n\n- Observed hosted failure: pip-audit run 96859346265 reported PYSEC-2026-3721 through pip 26.1.2.\n- Root-cause fix: refreshed the hash-locked CI dependency to pip 26.2.1 with both package hashes.\n- Exact head: 7787783\n- Exact base: 6a79f07\n- Local verification: 12 focused tests passed; pip-audit reported no known vulnerabilities; shell syntax and diff checks passed.\n- Hosted required checks and exact-head approval remain pending; decision remains WAIT_AND_REMEDIATE.

[seonghobae]

Current-head verification record

• Exact head SHA: f6c2404b1f32d98a97c10305bc651f2a989d6a4a
• Exact base SHA: fce028b4c3bf8e2e5e4819c1c5622e90cfa6ab39
• State: OPEN, non-draft, MERGEABLE, normal squash auto-merge enabled.
• Normal branch update completed from the former 6a79f075 base lineage.
• Final docstring coverage repair added the missing GitHubClient.__init__ docstring; the exact-head Strix/queue/coordinator suite now passes.
• Local verification at this exact head:
  • Strix/queue/coordinator tests — 112 passed, 16 subtests
  • bash -n scripts/ci/strix_quick_gate.sh — passed
  • actionlint .github/workflows/strix.yml — passed
  • python -m compileall — passed
  • interrogate --fail-under 100 scripts/ci — 100%
  • git diff --check — passed
• Unresolved non-outdated review threads: 0.
• The final push invalidated predecessor hosted evidence; current-head checks and qualifying approval remain pending.
• Decision: WAIT_AND_REMEDIATE pending current-head required checks and qualifying approval. No force-merge criteria are present.

[seonghobae]

Closing this competing Strix writer in favor of canonical owner PR #1153, without absorbing the proposed classifier.

Fresh exact-source review at f6c2404b1f32d98a97c10305bc651f2a989d6a4a shows is_model_quality_warning assembles MODEL QUALITY WARNING and the model-detail phrase with two independent whole-log greps. STRIX_LOG includes scan-target output, so target/repository text can supply those lines and cause a fallback classification. This violates the current provider-authentication boundary already enforced in #1153 for NVIDIA 404 and GitHub Models HTTP-410 evidence (anchored provider exception, required context, bounded same-line token).

The underlying model-quality incident remains fail-closed today: the warning is detected as infrastructure/failure evidence and cannot pass a required Strix gate. A retry optimization may be reintroduced only from authenticated provider-owned evidence that target output cannot spoof. The foreign pip-lock/docstring/scheduler assertions in this PR are already owned by #1198. No source is merged and no gate is weakened.