Skip to content

fix: env and importer reward integrity - #75

Merged
abhishekgahlot2 merged 3 commits into
mainfrom
fix/env-review
Oct 14, 2026
Merged

abhishekgahlot2 merged 3 commits into
mainfrom
fix/env-review

Conversation

@abhishekgahlot2

@abhishekgahlot2 abhishekgahlot2 commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Fixes reward-integrity issues in imported and built-in environments.

  • BigCodeBench: the runner checks the tests' top-level imports before loading the solution. A module the sandbox lacks now gives no verdict instead of scoring a correct solution 0.
  • EvalPlus: a task whose reference fails on any input is not published, and an earlier import's copy is removed. Before, those cases left the denominator, so a solution that failed them could score 1.0.
  • file_organizer: each README must hold its heading and description line. Heading-only files no longer earn 1.0.
  • cve_2021_44228_log_triage: only the answer's own fields are scored (keywords elsewhere in the output earn nothing). It needs the exact CVE ID and log4shell, counts indicators naming the jndi:ldap and jndi:dns lookups, and gives no remediation credit to a clause that holds a negation.

Each fix has a regression test that fails on main. Affected test files: 110 passed.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant