Skip to content

feat: import 16 OSWorld LibreOffice tasks as computer-use environments - #69

Merged
abhishekgahlot2 merged 19 commits into
DeepGym:mainfrom
Aling33:import-osworld
Oct 14, 2026
Merged

abhishekgahlot2 merged 19 commits into
DeepGym:mainfrom
Aling33:import-osworld

Conversation

@Aling33

@Aling33 Aling33 commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Closes #24.

Adds scripts/import_osworld.py, which imports 16 OSWorld LibreOffice tasks (13 Calc + 3
Writer, within the issue's 10-20 scope) as computer-use environments, one folder per task,
following the computer_use/signup_form layout.

Each generated folder

  • task.md — the task's own instruction, plus where the file is (open in Calc or Writer
    and at ~/Desktop/<file>) and to save with Ctrl+S.
  • setup.sh — decodes the staged assets/<file>.b64 to the Desktop and opens it in
    LibreOffice; fails loudly if soffice/libreoffice is missing, so a missing app is a
    setup error, not a silent zero. These tasks' configs are exactly a download plus an
    open, and that is what the script runs.
  • verifier.py — ports the task's own OSWorld evaluator to a state check over the saved
    file, never a screenshot:
    • Calc: the task's compare_table rules (sheet name / single cell / partial sheet
      data, OSWorld's read_cell_value semantics, pandas rounding) against the saved
      workbook;
    • Writer: exact text against the reference document.
    • Rules the untouched input already satisfies are guards, not progress checks: on
      one task a naive port paid 0.5 for changing nothing, and OSWorld's evaluator
      (all-or-nothing) can't tell a re-save from real work once a rule passes on the input.
      Guards make an untouched or deleted file score 0, while partial solutions earn the
      share of progress rules done.

Verification (offline, against OSWorld's real task configs and files)

  • The ported rule engine cross-checked cell-by-cell against OSWorld's own evaluator on
    the 16 real tasks: 1967 checks, 0 mismatches (including pandas' half-to-even
    rounding).
  • End-to-end through the real loader + verifier: all 16 score solved 1.0 / untouched
    0.0 / missing 0.0
    .
  • deepgym check passes on all 16 folders.
  • New tests/office_fixtures.py (stdlib-only .xlsx/.docx builders) and
    tests/test_import_osworld.py (32 tests) covering the importer, the evaluator port
    and the guards split.

Also in this PR: deepgym check on computer-use folders (the imports couldn't be
checked without it; it mirrors the skips deepgym qa makes since #68)

  • The files check no longer demands a reference_solution.* no desktop env ships; a
    shipped reference_actions.json stays optional — an agent drives the desktop.
  • The run checks report a skipped reference_passes when there is no episode to replay,
    or when the env needs a desktop (--mode daytona replays the built-ins' episodes),
    instead of failing over a reference a local session can't run.
  • lint_env reads a computer-use verifier the way the loader prepends it (_checks.py +
    PARAMS).

License — OSWorld is Apache-2.0; the imported task configs, instructions and files
keep its license (noted in the README section on the import).

Notes

  • The importer falls back to the GitHub contents API and hf-mirror.com where the usual
    raw hosts are blocked; the imported data was verified against a cached manifest.
  • The issue lists LibreOffice as part of the Daytona desktop; setup.sh still checks for
    the binary explicitly.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Aling33 and others added 15 commits October 14, 2026 18:04
Refs DeepGym#24.

scripts/import_osworld.py imports 16 OSWorld LibreOffice Calc and Writer
tasks as computer-use environments (task.md, setup.sh, verifier.py,
variants.json, metadata.json, assets/). OSWorld is Apache-2.0; the imported
configs, instructions and files keep its license.

The verifier ports each task's own evaluator into a state check over the
saved file — compare_table rules or a reference-text compare, never a
screenshot. The shared _checks.py reads workbooks (inline strings, merges,
number formats) and documents with the standard library only, and
cross-checks rule-by-rule against OSWorld's evaluator: 1967 checks, 0
mismatches. Rules the starting file already satisfies become guards, not
progress checks, so an untouched or vanished file scores 0 while a partial
solution earns the share of its rules it has done (OSWorld's evaluator was
all-or-nothing).

deepgym check now covers computer-use folders: it demanded a
reference_solution.* no desktop env ships and then replayed a reference a
local session can't run. It accepts the folder, and reports a skipped
reference_passes when there is no episode to replay or no desktop (the same
skips deepgym qa already makes); lint_env reads a computer-use verifier the
way the loader prepends it.

tests/office_fixtures.py builds .xlsx/.docx fixtures with the standard
library; tests/test_import_osworld.py covers the importer, the ported
evaluator and the guards split (32 tests).
…ch, and never equate a boolean with a number
@Aling33

Aling33 commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for taking this over and for the thorough pass — the reference episodes for every task and the hardened, bounded readers make this much stronger than my original. I'll stay off the branch; happy to pick up anything if useful.

@abhishekgahlot2
abhishekgahlot2 merged commit 634e52f into DeepGym:main Oct 14, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Import OSWorld tasks as computer-use environments

2 participants