Repository navigation
feat: import 16 OSWorld LibreOffice tasks as computer-use environments - #69
Merged
Merged
Conversation
abhishekgahlot2
force-pushed
the
import-osworld
branch
from
October 14, 2026 09:16
aa8ce34 to
4d475a3
Compare
Refs DeepGym#24. scripts/import_osworld.py imports 16 OSWorld LibreOffice Calc and Writer tasks as computer-use environments (task.md, setup.sh, verifier.py, variants.json, metadata.json, assets/). OSWorld is Apache-2.0; the imported configs, instructions and files keep its license. The verifier ports each task's own evaluator into a state check over the saved file — compare_table rules or a reference-text compare, never a screenshot. The shared _checks.py reads workbooks (inline strings, merges, number formats) and documents with the standard library only, and cross-checks rule-by-rule against OSWorld's evaluator: 1967 checks, 0 mismatches. Rules the starting file already satisfies become guards, not progress checks, so an untouched or vanished file scores 0 while a partial solution earns the share of its rules it has done (OSWorld's evaluator was all-or-nothing). deepgym check now covers computer-use folders: it demanded a reference_solution.* no desktop env ships and then replayed a reference a local session can't run. It accepts the folder, and reports a skipped reference_passes when there is no episode to replay or no desktop (the same skips deepgym qa already makes); lint_env reads a computer-use verifier the way the loader prepends it. tests/office_fixtures.py builds .xlsx/.docx fixtures with the standard library; tests/test_import_osworld.py covers the importer, the ported evaluator and the guards split (32 tests).
…ationship and compare dates as dates
…ocument never opens
…desktop is inconclusive
…r a crash or a timeout
…d, so an overrun scores 0
…ps waiting for it
…ch, and never equate a boolean with a number
…ooks whose declarations disagree
…larations, and require a Word manifest
abhishekgahlot2
force-pushed
the
import-osworld
branch
from
October 14, 2026 13:04
38a5118 to
3a87421
Compare
…readable, and read booleans as openpyxl does
… parse a document's related parts
Contributor
Author
|
Thanks for taking this over and for the thorough pass — the reference episodes for every task and the hardened, bounded readers make this much stronger than my original. I'll stay off the branch; happy to pick up anything if useful. |
…k unreadable, not empty
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #24.
Adds
scripts/import_osworld.py, which imports 16 OSWorld LibreOffice tasks (13 Calc + 3Writer, within the issue's 10-20 scope) as computer-use environments, one folder per task,
following the
computer_use/signup_formlayout.Each generated folder
task.md— the task's own instruction, plus where the file is (open in Calc or Writerand at
~/Desktop/<file>) and to save with Ctrl+S.setup.sh— decodes the stagedassets/<file>.b64to the Desktop and opens it inLibreOffice; fails loudly if
soffice/libreofficeis missing, so a missing app is asetup error, not a silent zero. These tasks' configs are exactly a download plus an
open, and that is what the script runs.
verifier.py— ports the task's own OSWorld evaluator to a state check over the savedfile, never a screenshot:
compare_tablerules (sheet name / single cell / partial sheetdata, OSWorld's
read_cell_valuesemantics, pandas rounding) against the savedworkbook;
one task a naive port paid 0.5 for changing nothing, and OSWorld's evaluator
(all-or-nothing) can't tell a re-save from real work once a rule passes on the input.
Guards make an untouched or deleted file score 0, while partial solutions earn the
share of progress rules done.
Verification (offline, against OSWorld's real task configs and files)
the 16 real tasks: 1967 checks, 0 mismatches (including pandas' half-to-even
rounding).
0.0 / missing 0.0.
deepgym checkpasses on all 16 folders.tests/office_fixtures.py(stdlib-only .xlsx/.docx builders) andtests/test_import_osworld.py(32 tests) covering the importer, the evaluator portand the guards split.
Also in this PR:
deepgym checkon computer-use folders (the imports couldn't bechecked without it; it mirrors the skips
deepgym qamakes since #68)filescheck no longer demands areference_solution.*no desktop env ships; ashipped
reference_actions.jsonstays optional — an agent drives the desktop.reference_passeswhen there is no episode to replay,or when the env needs a desktop (
--mode daytonareplays the built-ins' episodes),instead of failing over a reference a local session can't run.
lint_envreads a computer-use verifier the way the loader prepends it (_checks.py+PARAMS).License — OSWorld is Apache-2.0; the imported task configs, instructions and files
keep its license (noted in the README section on the import).
Notes
hf-mirror.comwhere the usualraw hosts are blocked; the imported data was verified against a cached manifest.
setup.shstill checks forthe binary explicitly.
Need help on this PR? Tag
@codesmith-botwith what you need.