Repository navigation
CI: prepare Windows wheel dependencies once and back off on upstream rate limits - #91
Open
olivhoenen wants to merge 3 commits into
Open
olivhoenen wants to merge 3 commits into
olivhoenen wants to merge 3 commits into
Conversation
All five Windows wheel jobs previously restored the same mutable win-deps cache key and then independently ran vcpkg against the same upstream source hosts; a cold matrix made five concurrent fetches of the same archives, and gitlab.dkrz.de rate-limited libaec with HTTP 429, failing the whole blocking wheel workflow. The five always-save steps also raced on the same immutable cache key. Restructure per issue #44's preferred design: - New windows_deps job prepares the pinned x64-windows dependency closure once, caching both C:\vcpkg\installed and the content-addressed C:\vcpkg\downloads under a key derived from the vcpkg manifest, the install script, the triplet, the UDA ref, and the runner's vcpkg baseline commit. It is the only job that ever saves the cache. - Windows wheel builds move to a dedicated build_wheels_windows job (needs: windows_deps) restoring that cache read-only with fail-on-cache-miss, so Linux/macOS legs are not delayed and no matrix job writes a cache key. - ci/wheels/cibw_before_build_win.sh reuses the UDA source archive from the cached downloads directory and skips the UDA build when the restored tree already provides uda-cpp.pc, so warm wheel jobs make no upstream fetches at all. - UDA_REF is pinned explicitly to 2.8.1 for Windows — the script's default, which Windows builds have always used (matrix.UDA_REF was never exported to the Windows build environment). - Both publish jobs now also depend on build_wheels_windows. Closes #44 (cherry picked from commit 7ba5c54)
First run of the windows_deps job (issue #44) confirmed the fix shape — exactly one job fetched upstream and the five wheel jobs skipped instead of failing five ways — but gitlab.dkrz.de rate-limited even the single libaec fetch, and vcpkg's own retries are seconds apart. Retry the closure install up to 5 times with growing (90s..450s) backoff; the script is idempotent, so each attempt resumes where the last stopped. Also quote $GITHUB_OUTPUT in docs.yml — the pre-existing shellcheck SC2086 that made the first actionlint run (issue #43) red. (cherry picked from commit 29e5136)
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
olivhoenen
requested review from
SimonPinches and
prasad-sawantdesai
as code owners
October 14, 2026 06:38
This was referenced Oct 14, 2026
| # The before-build script's default (matrix.UDA_REF was never exported to | ||
| # the Windows build environment, so Windows has always built this | ||
| # version); pinned explicitly here because it is part of the cache key. | ||
| UDA_REF: "2.8.1" |
Contributor
There was a problem hiding this comment.
Any particular reason for downgrading?
Suggested change
| UDA_REF: "2.8.1" | |
| UDA_REF: "2.9.3" |
| python -c "import sys, pprint, cibuildwheel; pprint.pp(sys.path); pprint.pp(cibuildwheel)" | ||
| python -m cibuildwheel --output-dir wheelhouse --config-file pyproject.toml | ||
| env: | ||
| UDA_REF: "2.8.1" # keep in sync with windows_deps.env.UDA_REF |
Contributor
There was a problem hiding this comment.
Should this also be bumped to
Suggested change
| UDA_REF: "2.8.1" # keep in sync with windows_deps.env.UDA_REF | |
| UDA_REF: "2.9.3" # keep in sync with windows_deps.env.UDA_REF |
| CIBW_CONFIG_SETTINGS: > | ||
| cmake.define.AL_BACKEND_HDF5=ON | ||
| cmake.define.AL_BACKEND_MDSPLUS=OFF | ||
| cmake.define.AL_BACKEND_UDA=OFF |
Contributor
There was a problem hiding this comment.
I think we can enable the UDA backend with Windows as done on Conda forge:
Suggested change
| cmake.define.AL_BACKEND_UDA=OFF | |
| cmake.define.AL_BACKEND_UDA=ON |
Contributor
There was a problem hiding this comment.
There may be path problems but we can fix them in follow-up commits once the CI build has been attempted.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the split of #85 (Tests and Benchmarks for regression) into smaller, single-purpose PRs that are easier to review.
Windows wheel jobs each ran vcpkg against the same upstream hosts and raced to save one cache key, so a single HTTP 429 (e.g. from gitlab.dkrz.de for libaec) failed the whole workflow. A new
windows_depsjob now prepares the pinned vcpkg dependency tree (C:\vcpkg\installedanddownloads) once, under a content-addressed cache key, retrying with backoff; the Windows wheel jobs restore it read-only. Also quotes$GITHUB_OUTPUTindocs.yml. The last commit (mine) aligns the new Windows job with the upstreamsetup-python@v7bump; the conflict with upstream's cibuildwheel 4.2.0 and setup-python v7 bumps was resolved by keeping the upstream versions.Fork history: yohannmarguier#44
Stack position: independent, based on
develop.Authorship: these commits are @yohannmarguier's, cherry-picked unchanged from his fork with
git cherry-pick -x, so each one keeps its original author, date and message plus a(cherry picked from commit …)link back to the fork. Only conflicts with currentdevelopand with the new commit order were resolved.🤖 Generated with Claude Code