feat: Python standalone images from python-build-standalone - #2154
feat: Python standalone images from python-build-standalone#2154reneleonhardt wants to merge 9 commits into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
|
At a highlevel:
|
Removed.
Explained above. PBS linux builds statically bundle zlib/bz2/sqlite3/OpenSSL/libffi/lzma/ncurses; the tarball binaries reference only glibc core libs, so the image adds zero python-specific deb packages. The 502-module stdlib import walk (ssl, sqlite3, zlib, ctypes, https) passes in-image with none; deliberate skips (GUI,
Done. Matrix is 3.13 + 3.14;
Included: |
|
I'm sorry I will get to this soon, lots of other things on my plate. |
|
No stress, my awesome contribution everyone's been waiting for isn't running away 😄 |
|
I'm no python expert, but the ldconfig seemed somewhat important to people. is CDLL sufficient in this case? are people not going to use "find_library()" ? |
|
I had a teammate recommend we keep the ldconfig binary and continue to generate the ldconfig.so.cache to allow find_library to continue to work |
be7c93f to
3c51b57
Compare
|
Good catch! Now python3/ parity.
Verified in the image: Scope note (deep-audited against the python3 image): PBS statically links zlib/bz2/sqlite3/OpenSSL/ffi/lzma/ncurses INTO libpython, so those modules work with no shared object to find - |
loosebazooka
left a comment
loosebazooka
left a comment
There was a problem hiding this comment.
I have some minor concerns as to how scanners will pick up the PBS embedded native libraries. If one were to scan the python 3.14 image, what happens when the embedded zlib has a CVE? Is its version discoverable?
|
|
||
| ## Provenance | ||
|
|
||
| Python comes from python-build-standalone release `20260807` |
There was a problem hiding this comment.
I think this part is a bit odd, and will have to be updated on every update. I think just linking to astral's pbs should be good. But not a specific version.
There was a problem hiding this comment.
Done — Provenance links to astral's PBS, no version pin. Exact pin lives in //private/extensions:python.bzl, managed by the updater.
| @@ -0,0 +1,335 @@ | |||
| set -o pipefail -o errexit -o nounset | |||
There was a problem hiding this comment.
gemini seems to think there's an issue with this script where a release tag change will go unnoticed 20260814 -> 20260815 which may be due to underlying library native library changes while the python version remains unchanged.
There was a problem hiding this comment.
Fixed. Was comparing patch-only; now compares the full patch+release pin, so a tag-only bump triggers an update. It was live: branch sat at 20260807 while 20260814 (expat 2.8.2 -> 2.8.3, same 3.13.15/3.14.7) was current — old logic said NO_CHANGE. Re-pinned to 20260814. Hermetic contract test covers tag-only bump, patch bump, new-minor fill, convergence.
I didn't mentioned this eariler, but these images would be considered new and will be categorized "experimental" for now. We don't have to solve the sbom/scanner problem now. I think we can try to get pbs themselves to produce and sbom and augment ours with theirs -- but none of this should block this PR or doing builds of these images. |
Good... because trivy couldn't find a way to dissect pbs, embedded native libraries keep being embedded 😄 |
New python/ package: distroless Python images from python-build-standalone (PBS) prebuilt tarballs instead of the Debian python packages used by python3/. PBS statically links the native libraries (OpenSSL, zlib, sqlite3, libffi, bz2, lzma, ncurses, expat) into the interpreter, so the image adds no python runtime debs - only libc-bin, whose ldconfig plus a generated per-arch ld.so.cache keeps ctypes.util.find_library() working (python3/ parity). Matrix: 3.13.15 / 3.14.7 x debian13 x amd64/arm64/s390x/riscv64 x root/nonroot x normal/debug - more Python versions than Debian ships on any single release (trixie has 3.13 only) and an arch python3/ never had (s390x); 3.15 is added by the updater once PBS releases it stable. debian12 and armv7/ppc64le are excluded (PBS publishes no usable builds). Pinned to PBS release 20260814.
…chives) knife update-python-archives keeps the PBS pins in //private/extensions:python.bzl current: reads the matrix (minors + archs) from python/config.bzl, queries the latest PBS release, rewrites every archive (url/sha/version), and - when a new stable minor appears - extends the config.bzl matrix, MODULE.bazel use_repo and the version testdata. An update triggers on a patch move or on a release-tag move with unchanged patches (PBS rebuilds its bundled native libraries, e.g. expat, under the same CPython version). Idempotent; --verbose/--dry-run for inspection. Tests: hermetic smoke test //python:update_python_archives_test (parser consistency) + black-box contract test //python:update_python_archives_contract_test running the updater offline against fixture releases - tag-only bump, patch bump, new-minor fill, convergence.
The update now also regenerates an SPDX 2.3 SBOM from the release's component manifest (pythonbuild/downloads.py): cpython versions, bundled pip/setuptools and the statically embedded native libraries, each with version, license, sha256 and download URL. Build-time-only tools are excluded. Covered by the contract test (SBOM regenerated on tag/patch/new-minor updates, stable on no-op runs).
The SBOM entries now carry purls: pkg:pypi for pip (scannable by trivy, license MIT from the pypi fallback map) and pkg:generic for the native libraries. Versions prefer the manifest's actual_version (sqlite shipped its SQLITE_VERSION_NUMBER, 3530100, instead of 3.53.1.0), and setuptools is no longer listed: install_only tarballs do not ship it, it is a PBS build-time tool.
…RED) A PBS release can bump its statically linked native libraries while the CPython versions stay unchanged (maintainer-reported gap: python pin alone would not notice). The updater now downloads the x86_64 install tarball (sha-verified against SHA256SUMS), dissects libpython3*.so and compares the embedded openssl/zlib/expat/ncurses/bzip2/sqlite/xz version markers against the release manifest; a mismatch aborts the update. trivy cannot see these libraries in any mode, so the dissection is the only drift check possible. The contract test gained a hermetic tarball fixture (PBS_TARBALL_FILE) and a drift phase: a tarball whose markers no longer match the manifest must fail the updater without mutating the workspace. get_python_archs now returns an empty result for a minor missing from config.bzl instead of aborting under errexit+pipefail (the new-minor fallback relies on it).
pbs_cve_check.py queries the NVD API for the dissection-verified versions of the seven embedded libraries (openssl/sqlite/zlib/expat/bzip2/ncurses/xz) and aborts the update on HIGH/CRITICAL CVEs; dependency-reference false positives (mutt/OpenLDAP-style CPE matches with vulnerable=false) are filtered out. trivy has no advisory feed for pkg:generic, so this is the only real CVE signal for those libraries. NVD_API_KEY removes the rate-limit sleeps, PBS_SKIP_CVE_CHECK=1 bypasses. Hermetic test with fixture NVD responses covers the clean path, the RED path and the false-positive filter. Live result for the 20260814 pin: sqlite 3.53.1 carries CVE-2026-11822/CVE-2026-11824 (fixed in 3.53.2); the rest are clean.
7b9584b to
f205dd4
Compare
2 participants
Summary
New
python/package (mirrorsnodejs/) building distroless Python images from python-build-standalone (PBS) prebuilt tarballs instead of Debian-packaged python. The image base stays Debian (//ccprovides glibc); only the python distribution method changes.Matrix: 3.13.15 / 3.14.7 x debian13 x amd64 / arm64 / s390x / riscv64 x root/nonroot x normal/debug.
Tags:
python3.13-debian13,python3.14-debian13(+ per-arch and default-distro variants, nodejs publish pattern).Why PBS
knife update-python-archives.Platform exclusions (verified, not oversights)
/lib/ld-linux.so.3); the distroless armhf base ships/usr/lib/ld-linux-armhf.so.3- the binary cannot exec (confirmed in Docker). Removed./usr/lib/ld64.so.1,/usr/lib/ld-linux-riscv64-lp64d.so.1).Notes
find_library()works for system libs (c/ssl/z verified): image ships/sbin/ldconfig(libc-bin) + a per-archld.so.cache(python3/ parity, maintainer request); regenerated viabazel run //python:update_ldconfig, checked by//python:check_ldconfig_tests(4/4) + a CST test. Statically-linked PBS libs have nothing to find;python -m venvworks (ensurepip wheels bundled);cryptmoot (removed in 3.13).python3/(deb) untouched.Verification
bazelisk build //python/...green (debian13, 3.13+3.14, 4 archs).3.14.7 x86_64, ssl certs OK.//python:check_ldconfig_tests4/4 (amd64/arm64/s390x/riscv64); CST includes a find_library test; CDLL(soname) works (libc/libssl/libgcc/libpython).knife update-python-archives): no-op idempotent;--verbose/--dry-run; fills new stable minors, drops EOL tail, sorts+dedupes the matrix; hermetic smoke test//python:update_python_archives_test; pre-flight validation (RED mutates nothing; recovery via git, lockfile reset documented).kind(oci_image, deps(:sign_and_push)); CST taggedmanual+arch run viaattr(tags, "amd64", ...)— same rails as python3/;check-ldconfig.yamlextended to cover//python:check_ldconfig_tests.Checklist
bazelisk build //python/..., structure tests