feat(docker): provision ROUGE scorer deps and corpora in dev image - #507
Merged
Merged
Conversation
Signed-off-by: Rashid Kaleem <[email protected]>
Signed-off-by: Rashid Kaleem <[email protected]>
Signed-off-by: Rashid Kaleem <[email protected]>
…467) The published ghcr.io/mlcommons/endpoints:*-livecodebench image inherited zstd-compressed base layers from the dhi.io Docker Hardened Image base. enroot/ pyxis does not route the docker-namespaced zstd media type to a decompressor and hands the raw blob to tar, failing with "tar: This does not look like a tar archive" (#467). - scripts/lib_registry.sh: assert_gzip_layers — inspects the pushed manifest and fails on any non-gzip layer (walks a manifest list; docker + oci media types). - scripts/push_docker_image.sh: push via --output type=image,push=true,compression=gzip,force-compression=true so base layers are re-compressed to gzip; verify after push. - livecodebench/push_image.sh: verify layers after both the --platform and native push paths. verified a zstd build reproduces the tar error at pyxis import; the force-compression gzip build imports and runs cleanly.
Signed-off-by: arekay-nv <[email protected]>
…guards
Follow-up hardening for the image push/pull tooling (client + lcb-service):
- lib_registry.sh: add ref_exists_in_registry, a fail-closed 0/1/2 registry
existence probe. A two-stage match screens tooling/credential/permission
errors to "indeterminate" BEFORE matching not-found phrasings, so a missing
docker-credential helper ("executable file not found") can't be mistaken for
an absent tag (which would let a push overwrite an immutable one). Handles
GHCR "<ref>: not found" and ECR "name unknown ... does not exist".
- push_docker_image.sh: add --force; refuse to overwrite an existing :<sha>
unless --force. Blocks on any non-absent probe result (fail closed).
- push_image.sh: --no-build now verifies layers on a transient staging tag then
promotes onto the pinned tag only on success (verify-before-publish, so a
zstd/#467 image never poisons the immutable pin). Promote with
--prefer-index=false and re-assert the pin. Immutability guard reuses the
shared ref_exists_in_registry.
- _image_env.sh: always append an idempotent "-livecodebench" suffix to the LCB
tag so the image is self-identifying and never collides with the client
image's bare :<sha> in a shared registry package. Applied in the shared
push/pull helper, so pull stays symmetric (consumers still pass the SHA).
- Dockerfile.dev / lcb_serve.dockerfile: add OCI title/description LABELs so
docker inspect self-identifies each image. Client label placed last (cache);
its description is capability-neutral (PROVISION_DSR1/PROVISION_VBENCH may be 0).
- README: document the tag scheme and add a maintainer recipe to publish the
official ghcr.io/mlcommons/endpoints image.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Address review-council findings on the image publish tooling: - lib_registry.sh: ref_exists_in_registry no longer classifies auth-hidden-as- not-found as absent. Registries that answer an unauthorized/private repo with "repository does not exist or may require 'docker login': denied" previously matched the not-found phrasing and returned "absent" (rc=1), letting a push- only / split-scope credential overwrite an immutable tag. Auth/denied/401/403 phrasings are now screened to indeterminate (rc=2) BEFORE the not-found match; GHCR "<ref>: not found" and ECR "name unknown" still resolve to absent. - publish-image.yml: add a `force` workflow_dispatch input (forwarded as a fixed --force flag via env, never interpolating the input into the command) so a SHA can be re-published (e.g. a different arch, or a failed run); document that platform/DSR1/cache are not part of the tag. - push_docker_image.sh / push_image.sh: document that the buildx-path assert_gzip_layers is a post-publish confirmation (the build forces gzip, so the pushed image is enroot-safe by construction), and that a transient inspect flake fails the job after the tag is live (re-run with --force). Reword the client guard comment: the :<sha> gate covers the whole build (both tags). - README: correct the pinned tag to :<sha>-livecodebench; add an official pull example (LCB_IMAGE_NAME=endpoints, else it resolves to a different repo); document that --no-build stages+verifies+promotes, is host-arch only, ignores --platform, and rejects (does not repair) zstd local images. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
push_docker_image.sh forwarded only PROVISION_DSR1, so PROVISION_VBENCH=0 was silently ignored and the VBench (WAN 2.2) scorer was always baked in. Forward PROVISION_VBENCH as a build-arg symmetrically with PROVISION_DSR1, and expose a provision_vbench choice input in the publish-image workflow. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Install the `rouge` extra in both uv sync layers and prefetch the scorer's
runtime data, so a pushed image can run `eval_method: "rouge"` with no
network - matching how PROVISION_DSR1 / PROVISION_VBENCH already make the
DeepSeek-R1 and VBench evaluators self-contained.
The extra is declared explicitly rather than inherited from `test`, since
ROUGE is a runtime capability of the published image and should not depend
on a test-only extra to supply it.
Installing the packages is not enough on its own: nltk.sent_tokenize needs
the punkt/punkt_tab corpora and evaluate.load("rouge") fetches its metric
script from the HF Hub. A new PROVISION_ROUGE=1 block prefetches both and
then asserts the corpora resolve, so a silent download failure fails the
build instead of the benchmark.
Both caches land in their default per-user locations (~/nltk_data,
~/.cache/huggingface). Deliberately not redirected to /opt via
NLTK_DATA/HF_HOME: HF_HOME is global at runtime, so repointing it would
orphan what the DSR1/VBench stages already cached under the default path.
The block sits after VBench and before the LABEL layer, so editing it
rebuilds neither the heavy evaluator stages nor anything but image config.
Requires the `rouge` extra from #506 - merge that first, or `uv sync
--extra rouge` fails here.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
github-actions
Bot
added
the
size/normal
attafosu
approved these changes
Sep 16, 2026
Collaborator
|
@anandhu-eng do we want to retarget this to main since the arekay/push_image_workflow branch has been merged? |
anandhu-eng
changed the base branch from
arekay/push_image_workflow
to
main
September 30, 2026 04:49
arekay-nv
approved these changes
Oct 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
rougeextra in bothuv synclayers, so the published image can runeval_method: "rouge". Declared explicitly rather than inherited fromtest— ROUGE is a runtime capability of the image and shouldn't depend on a test-only extra.PROVISION_ROUGE=1block prefetching the scorer's runtime data, matching howPROVISION_DSR1/PROVISION_VBENCHalready make their evaluators self-contained.Installing the packages isn't enough on its own:
nltk.sent_tokenizeneeds thepunkt/punkt_tabcorpora andevaluate.load("rouge")fetches its metric script from the HF Hub. The block prefetches both, then asserts the corpora resolve — a silent download failure fails the build rather than the benchmark.Two deliberate choices
Caches stay in their defaults (
~/nltk_data,~/.cache/huggingface) rather than/optviaNLTK_DATA/HF_HOME.HF_HOMEis global at runtime, so repointing it would orphan whatever the DSR1/VBench stages already cached under the default path.Placed after VBench, before
LABEL— editing it rebuilds neither the heavy evaluator stages above nor anything but image config below.Verified
A full build of this file isn't possible until #506 merges, so the block was validated in an isolated
python:3.12.11-slimprobe image running as a non-root user with the identical commands:rouge metric cached+punkt corpora ok.docker run --network=nonecomputesrougeLfrom the baked cache — the offline claim, confirmed end to end.PROVISION_ROUGE=0correctly skips the block.Probe images were removed afterward. The prefetch commands were also run against the real
rouge-synced venv, including an offline replay.🤖 Generated with Claude Code