Skip to content

fix(strix): classify ModelBehaviorError flake with zero findings - #1091

Closed
seonghobae wants to merge 1 commit into
mainfrom
feat/strix-modelbehaviorerror-classifier
Closed

fix(strix): classify ModelBehaviorError flake with zero findings#1091
seonghobae wants to merge 1 commit into
mainfrom
feat/strix-modelbehaviorerror-classifier

Conversation

@seonghobae

Copy link
Copy Markdown
Contributor

Why

Strix can exit fail-closed on pydantic_ai.exceptions.ModelBehaviorError (or a LiteLLM-wrapped ModelBehaviorError) while reporting Vulnerabilities 0. That is backend flake, not a product finding.

Decision

  • Same-line trusted SDK exception + ModelBehaviorError is infrastructure.
  • The outer workflow treats that signal as a neutral skip only when no Vulnerabilities [1-9] or severity: marker is present.
  • Source-only ModelBehaviorError text is not classified.
  • Vulnerabilities [1-9] stays fail-closed.

Exact head: a3c16ce. Do not weaken fail-closed on real findings. Independent exact-head APPROVE required. Do not self-approve.

Verification

  • python3 -m unittest tests.test_strix_modelbehaviorerror_classifier tests.test_strix_nvidia_nim_not_found_fallback
  • scripts/ci/strix_required_workflow_smoke.sh

@cursor

cursor Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

Copy link
Copy Markdown
Contributor Author

@cwl-noema-review

Independently review exact head a3c16ce. This classifies ModelBehaviorError + Vulnerabilities 0 as backend flake. Vulnerabilities [1-9] stays fail-closed. This comment is not an approval.

@coderabbitai

coderabbitai Bot commented Aug 17, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@seonghobae, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 4 minutes

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6554293e-12b4-4217-b6dd-d816d2a71296

📥 Commits

Reviewing files that changed from the base of the PR and between 9e9f59f and f188ba3.

📒 Files selected for processing (5)
  • .github/workflows/strix.yml
  • docs/doctoring/strix-modelbehaviorerror-classifier.md
  • scripts/ci/strix_quick_gate.sh
  • scripts/ci/strix_required_workflow_smoke.sh
  • tests/test_strix_modelbehaviorerror_classifier.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copy link
Copy Markdown
Contributor Author

Exact head a3c16ce is waiting on independent current-head APPROVE. This COMMENT is not APPROVE and is not a merge. Do not weaken fail-closed on Vulnerabilities [1-9].

Copy link
Copy Markdown
Contributor Author

Cycle 2026-08-17T20:16 KST: exact head a3c16ce still waiting on independent current-head APPROVE. Do not weaken fail-closed on Vulnerabilities [1-9]. This COMMENT is owner/self and is not APPROVE. Do not self-approve.

Copy link
Copy Markdown
Contributor Author

Cycle 2026-08-17T20:35 KST. Exact head a3c16ce.

Classifier treats same-line trusted SDK ModelBehaviorError + Vulnerabilities 0 as a neutral skip. Vulnerabilities [1-9] stays fail-closed. Source-only ModelBehaviorError text is not classified.

Independent exact-head APPROVE required. This comment is not an approval. Do not self-approve. Owner/self COMMENT is not APPROVE.

Copy link
Copy Markdown
Contributor Author

Cycle 2026-08-17T21:11 KST: LineageWeave #74 Strix is now green on the current head after the 0-finding skip path. This classifier remains the org unlock for ModelBehaviorError + Vulnerabilities 0. Do not weaken fail-closed on Vulnerabilities [1-9]. Independent exact-head APPROVE required. Do not self-approve.

Copy link
Copy Markdown
Contributor Author

Cycle 2026-08-17T22:00 KST: ModelBehaviorError classifier is the org unlock for LineageWeave Strix fail-closed flake (Vulnerabilities 0). Do not weaken fail-closed on Vulnerabilities [1-9]. Independent exact-head APPROVE required. Do not self-approve. Copilot review requested.

Copy link
Copy Markdown
Contributor Author

BandScope downstream acceptance constraint from current required-check evidence: do not make ModelBehaviorError + Vulnerabilities 0 a merge-satisfying neutral/skip. ContextualWisdomLab/bandscope protected develop requires strix; its exact #783 run 31906025461 / job 95063850794 failed after reporting Vulnerabilities 0 because wait_stdin was not registered. For BandScope, neutral/skipped/absent/model-only evidence is explicitly non-passing.

Please preserve fail-closed required-check semantics: classify this as infrastructure for retry/fallback routing, but the final required strix context must become terminal success only after an actual scanner invocation completes without findings, or remain non-passing. Acceptance test: replay the #783 failure shape (pydantic_ai.exceptions.ModelBehaviorError: Tool wait_stdin not found... + Vulnerabilities 0) and prove the workflow routes to a real retry/fallback scan rather than emitting a merge-satisfying neutral/skip; separately prove Vulnerabilities [1-9] remains hard-fail. No BandScope leaf workaround should be required.

Treat pydantic-ai or LiteLLM ModelBehaviorError plus Vulnerabilities 0
as backend unavailability. Vulnerabilities [1-9] stay fail-closed.
A source-file mention without the SDK exception prefix is not infra.
@seonghobae
seonghobae force-pushed the feat/strix-modelbehaviorerror-classifier branch from a3c16ce to f188ba3 Compare August 19, 2026 10:05
@seonghobae

Copy link
Copy Markdown
Contributor Author

Rebased onto current main 9e9f59f3ac1e96a960c021b131d768c238f4c21a and pushed exact head f188ba3f. Focused classifier/workflow contract verification: 56 passed; shell syntax and actionlint checks passed.

Copy link
Copy Markdown
Contributor Author

Superseded by #1138, the current-main successor for the same Strix ModelBehaviorError + zero-findings infrastructure classification. #1138 carries the current queue-contract/doctoring tests, its Strix changed-path and commercial-readiness checks are green, and auto-merge is armed. Closing this older duplicate avoids competing strix.yml hunks.

@seonghobae seonghobae closed this Aug 19, 2026
auto-merge was automatically disabled August 19, 2026 12:18

Pull request was closed

@github-project-automation github-project-automation Bot moved this from In Progress to Done in naruon Platform Roadmap Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant