Skip to content

fix: reject unknown attacks and invalid sub-scores, drop stale QA results, keep export runtimes and initial observations - #76

Merged
abhishekgahlot2 merged 1 commit into
mainfrom
fix/qa-provenance-review
Oct 14, 2026
Merged

abhishekgahlot2 merged 1 commit into
mainfrom
fix/qa-provenance-review

Conversation

@abhishekgahlot2

@abhishekgahlot2 abhishekgahlot2 commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor
  • QA gate: unknown attack strategies are rejected before any attack runs (422 over the API). A misspelt one ran a no-op and read as robust.
  • QA gate: an audit drops a changed verifier's stored result before its retest, so an inconclusive retest no longer leaves the old result standing.
  • Invalid case scores and NaN or infinite reward components no longer become a trainable 0 under a valid overall score. The verifier wrapper checks them like the overall score: invalid after the solution ran is a judged 0 (solution_error); invalid from the verifier alone is invalid_score with no verdict. The host rejects them too (InvalidScoreError).
  • OpenRLHF: a sample whose verifier failed or returned an invalid result gets infra_failure_score instead of 0. Judged model failures and runs still going at the request deadline still score 0.
  • Harbor export refuses snapshot-backed environments instead of exporting them on a plain image.
  • RL trace rows carry initial_observation and initial_screenshot.

Each fix has a regression test that fails on main.

…ults, keep export runtimes and initial observations
@abhishekgahlot2
abhishekgahlot2 force-pushed the fix/qa-provenance-review branch from 029650c to 678c5cf Compare October 14, 2026 08:33
@abhishekgahlot2
abhishekgahlot2 merged commit 0bdec86 into main Oct 14, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant