Skip to content

feat: add secure OJBench alpha adapters - #4

Merged
aabbdev merged 2 commits into
mainfrom
research/ojbench-dmoj-20260806
Aug 7, 2026
Merged

feat: add secure OJBench alpha adapters#4
aabbdev merged 2 commits into
mainfrom
research/ojbench-dmoj-20260806

Conversation

@aabbdev

@aabbdev aabbdev commented Aug 7, 2026

Copy link
Copy Markdown
Owner

Summary

  • add separate ojbench_python and ojbench_cpp alpha tasks with the official 232-problem language tracks and eight epochs for Pass@1/Pass@8
  • pin the official prompt/testdata revision, OJBench source commit, DMOJ judge commit, base image, and Python runtime dependencies
  • download only the selected problem's LFS archive and validators into a host cache mounted read-only in the judge
  • run submissions through DMOJ cptbox in a network-disabled, read-only, bounded Docker sandbox and keep per-case tests/results out of Inspect logs

Security boundary

The container drops all capabilities, restores only SYS_PTRACE as required by DMOJ to supervise its child, uses no-new-privileges, disables networking, and bounds memory/PIDs. Generated programs remain under DMOJ's syscall and filesystem policies; the mounted /problems tree is not readable by submissions.

A malicious Python submission attempting to read /problems/NOI/loj-2083/init.yml received DMOJ IR rather than test content.

Validation

  • pre-commit run --all-files
  • pytest — 474 passed, 11 skipped
  • targeted mypy — success across all OJBench modules
  • real arm64 DMOJ self-tests — CPP17 and PYPY3 passed
  • real loj-2083 judging — Python and C++ wrong submissions returned WA over 20 cases
  • lazy post-start download — newly cached loj-3758 became visible through the existing read-only bind mount
  • Docker Compose profile inspection — network none, read-only rootfs, no-new-privileges, 3 GiB memory, 128 PIDs
  • wheel and sdist — both contain the OJBench Dockerfile and compose definition

Alpha gates

The complete 7.85 GB problem corpus has not yet been exercised across all custom validators, and no credentialed model Pass@1/Pass@8 run has been completed. Both registry IDs intentionally remain alpha until those broader validation gates pass.

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

✅ Benchmark documentation has been automatically updated.

@aabbdev
aabbdev merged commit 733f87f into main Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant