Repository navigation
feat: agent-facing contracts: docstring index, run headers, verified doc commands - #32
Merged
Merged
Conversation
added 6 commits
October 14, 2026 09:24
…t trials
Running the library from documentation alone exposed behavior that did not match what the docs
promised. Each fix below failed a test before it was made.
- Hydra's run directory was pinned to ./output and ignored output_root, so any command failed with
FileExistsError where ./output was not creatable, including mode=help
- set_cfg assigned a full config's overrided_cfg: {} back onto the instance, dropping the records of
fields processed before it and making mode=help under-report overrides
- exp_class no longer shows up as an override; store_and_run_exp fills it in
- examples raise UnknownExperimentModeError instead of NotImplementedError, so an unsupported mode
lists the ones that are handled
- vit_tp builds the train dataloader only when a batch exists, instead of training zero steps and
failing with ZeroDivisionError
- tinyexp-run-with-redis refuses a launcher=ray command, which already owns its Redis, and resolves
-m the way the child process does (torchrun -m and accelerate launch -m included)
The evaluation mode is renamed from val to eval in the mnist and resnet examples, matching vit_tp.
mode=val is not kept as an alias and now reports the supported modes.
An agent that only reads the docs has to be able to call the library correctly, and the first command it runs has to work. - every public module, mixin and accelerator carries a docstring: responsibility, exports and a minimal usage example. The API page renders all of them instead of only the top-level package - AGENTS.md gains a task-to-file index, the places not to explore, the order of preference for a change, and what mode=help does and does not check - SKILL.md becomes the same index for an installed package, where AGENTS.md and docs/ are absent - every example opens with a run header (Modes, Run, Env, Needs, Out), and the new tinyexp/examples/README.md maps requirements to examples - Redis, W&B, sampling and logging helpers describe their units, defaults and lifecycle tests/test_tinyexp_artifacts.py asserts that import tinyexp stays free of torch and wandb.
The documentation is the interface an agent works from, so it is now checked the same way code is. - every command in a run header, README, SKILL and docs/*.md is validated against the config it targets (unknown keys and value types are rejected); example lists are discovered automatically, so a new example is covered without touching the test. The check is static: a command can be legal yet no longer describe what the example does - every Modes value in a run header must be handled in the example's run() - tests/tine_engine (a typo, referenced only by AGENTS.md) becomes tests/tiny_engine - tinyexp/tiny_engine gains the missing __init__.py docstring
vit_tp_exp uses ``from __future__ import annotations``, so its config fields are annotated with strings such as "RayCfg". OmegaConf resolves them with typing.get_type_hints, which on Python 3.9 looks up names only in the module globals, not in the class body where the nested config classes live; Python 3.10 added the class namespace. Every vit_tp_exp command, mode=help included, failed on 3.9 with "NameError: name 'RayCfg' is not defined". The bug predates this branch; the new example smoke tests are the first to run vit_tp_exp on 3.9. The ten config fields now name their type as VitTpExp.<Cfg>, which resolves from the module globals on every supported version.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes the library usable from documentation alone: an agent that never reads the source can
define an experiment, run it on CPU/GPU/Ray, use checkpoints and the data helpers, and trust the
commands the docs show.
Three commits on
dev(3d6c09d):fix: 🐛 harden config, checkpoint and launcher edges found by the agent trialsdocs: ✏️ document every public contract so the source need not be readtest: 💍 check the documented commands and the example run headersmode=valis renamed tomode=evalin the mnist and resnet examples (vit_tp already usedeval).No alias is kept; the old value now fails with
Unknown mode 'val' ... train, eval, help.Names about the validation set are unchanged (
val_dataloader,val_batch_size_per_device, theImageNet
val/directory). The change is recorded in the first commit's message for the changelog.Also fixed (each had a failing test first)
./output, ignoringoutput_root: any command failed withFileExistsErrorwhere./outputwas not creatable, includingmode=helpset_cfgassigned a full config'soverrided_cfg: {}back onto the instance, dropping overriderecords and making
mode=helpunder-reportexp_classno longer appears as an overrideUnknownExperimentModeErrorinstead ofNotImplementedErrorZeroDivisionErrortinyexp-run-with-redisrefuses alauncher=raycommand (the driver owns Redis there) andresolves
-mthe way the child process does, includingtorchrun -mandaccelerate launch -mVerification
make check,make docs-test(strict) passimport tinyexpstays free of torch and wandb (subprocess assertion)README.md,SKILL.mdanddocs/*.mdis validatedstatically against the config it targets
Known limits
as commands and configuration only, not executed.