From 3051f128d42d8bdaa938b0906b9198c5852022a4 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 24 Aug 2026 21:28:02 +0000 Subject: [PATCH 1/2] smoke: add the CTI workspaces to the local runner, sharing one arcticpy recipe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit heart/smoke.py is the local mirror of CI, but the `smoke:` block in config/repos.yaml had no autocti entries and import_names had no PyAutoCTI — so neither CTI smoke suite could be run locally, and the only way to exercise them was to push and wait. That is the slowest possible loop for the repo group that just acquired new coverage (autocti_workspace#28 gave that repo its first CI). The design question was arcticpy: `import autocti` hard-requires it, it is not a pip dependency, and a composite action cannot be invoked from Python. Two of the three options in the task turn out to answer different questions, so both are taken: * WHERE THE RECIPE LIVES — the action's five run-steps are extracted into .github/actions/install-arcticpy/install_arcticpy.sh. The action calls it via ${{ github.action_path }} (a composite action is downloaded with its whole directory, so cross-repo consumers with no PyAutoHeart checkout still work); smoke.py calls the same file out of the Heart checkout. One file, one pin, both consumers running identical bytes. * WHAT TRIGGERS IT — a per-workspace `arcticpy: true` key in the `smoke:` block, mirroring the input the CTI CI callers already pass. Declared rather than inferred from `PyAutoCTI in chain`, so CI and the local runner are configured by the same explicit statement. Rejected: a Python leg in smoke.py mirroring the recipe. That is the divergence #170 was created to end. While extracting, found the pin had quietly acquired two more copies: action.yml's `version` input defaulted to "2.6", and arcticpy-action.yml passed `${{ inputs.version || '2.6' }}`. The latter matters — the self-test would have gone on proving the OLD version built after a bump. Both now defer to the script's single default; no consumer passes `version` at all. Two things found by running it rather than reading it: * The GSL probe used `ls a b c`, which exits non-zero when ANY operand is missing — so on every machine that actually has GSL (in exactly one prefix) it reported the headers absent and refused to build. Now tests each prefix independently, and the list is overridable via ARCTICPY_GSL_PREFIXES, which also serves the no-root workaround in PyAutoCTI/AGENTS.md. * `pip check` can NEVER pass in a CTI environment. arcticpy declares numpy~=1.21 and is installed with --no-deps on purpose, since honouring it downgrades numpy below 2.0 and breaks the stack, so the preflight reported "arcticpy 2.6 has requirement numpy~=1.21, but you have numpy 2.5.2" and destroyed every environment immediately after building it. The preflight now tolerates exactly that one line, and only for a workspace that declared `arcticpy: true`; any other broken requirement still fails. The local leg never runs apt: a dev command must not mutate system packages, and apt-get does not exist on macOS. It proves the headers are present and fails with the install line instead. The fingerprint hashes the shared script, so editing the recipe or bumping the pin invalidates the cached environment rather than silently reusing a stale one. Verified end to end, not just "the config parses" — `pyauto-heart smoke autocti_test` on Python 3.12 built its environment (arcticpy 2.6 from the shared recipe), passed preflight, and ran 3/3 scripts PASS. Refs #172 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014wiN7R1yaeGj6k1Pa1FEq4 --- .github/actions/install-arcticpy/action.yml | 123 +++----- .../install-arcticpy/install_arcticpy.sh | 153 ++++++++++ .github/workflows/arcticpy-action.yml | 7 +- config/repos.yaml | 21 ++ heart/smoke.py | 109 ++++++- tests/test_repo_config.py | 53 ++++ tests/test_smoke.py | 271 ++++++++++++++++++ 7 files changed, 643 insertions(+), 94 deletions(-) create mode 100755 .github/actions/install-arcticpy/install_arcticpy.sh diff --git a/.github/actions/install-arcticpy/action.yml b/.github/actions/install-arcticpy/action.yml index 7ed1810..218707f 100644 --- a/.github/actions/install-arcticpy/action.yml +++ b/.github/actions/install-arcticpy/action.yml @@ -4,48 +4,27 @@ description: >- hard-requires. The canonical recipe for the whole organism — every CTI repo's CI consumes this action rather than carrying its own copy. -# WHY THIS EXISTS +# The recipe, the reasoning behind its two awkward pip flags, and the single +# `arcticpy==2.6` pin all live in `install_arcticpy.sh` beside this file. This +# action is a thin CI wrapper over it — read that script, not this, to +# understand or change what gets installed. # -# arcticpy is deliberately NOT a pip dependency of autocti: -# -# * its PyPI distribution is a source-only C++ sdist — it needs libgsl-dev -# headers and a toolchain to build, and there is no wheel; -# * its own requirements downgrade numpy below 2.0, which breaks a modern -# PyAuto stack. -# -# So it is built with --no-build-isolation (reusing the numpy already present -# instead of resolving its own) and --no-deps (so it cannot drag numpy back -# down). Those two flags are what make the recipe fiddly, and each one costs -# something that has to be paid back explicitly: -# -# --no-build-isolation pip does NOT create an isolated build environment and -# does NOT read pyproject's build-system.requires, so -# every BUILD dependency must already be installed. -# arcticpy declares none of them. -# --no-deps pip installs no RUNTIME dependencies either, so the -# packages arcticpy imports at import time must also be -# installed by hand. -# -# Both sets are installed below, and the step ends by proving the result -# actually imports. -# -# Verified 2026-08-24 in a clean container against a bare venv: each missing -# build dependency failed the build naming the next one. Without setuptools the -# build dies at `BackendUnavailable: Cannot import 'setuptools.build_meta'` — -# and Python 3.12+ venvs no longer ship setuptools by default, which is why the -# recipe that omitted it was a real hazard rather than a style nit. -# -# This step is intentionally SELF-CONTAINED: it upgrades pip/setuptools/wheel -# itself rather than relying on a caller having done so in a preceding step. -# The previous arrangement had autocti_workspace_test's smoke_install.sh -# depending on a `pip install --upgrade pip setuptools wheel` line living in a -# different repository's workflow, with nothing stating the dependency. +# Why the split: `heart/smoke.py` prepares the same arcticpy for Heart's LOCAL +# smoke runner, and a composite action cannot be invoked from Python. A shell +# script is the one shape both consumers can execute, so the recipe stays in +# exactly one place. #170 replaced four divergent copies with one; a second copy +# here would undo it. inputs: version: - description: "arcticpy version to build. THE single pin for the organism — bump it here." + description: >- + Override the arcticpy version to build. Leave EMPTY (the default) to use + the organism's pin, which lives in exactly one place — the + ARCTICPY_VERSION default at the top of install_arcticpy.sh. Bump it there. + A default here would be a second copy of the pin, which is the thing #170 + removed. required: false - default: "2.6" + default: "" sudo: description: >- Use sudo for the apt-get leg. GitHub-hosted runners need it; a container @@ -67,59 +46,21 @@ outputs: runs: using: composite steps: - - name: Install GSL headers - if: ${{ inputs.install-gsl == 'true' }} - shell: bash - run: | - set -euo pipefail - SUDO="" - if [ "${{ inputs.sudo }}" = "true" ]; then SUDO="sudo"; fi - $SUDO apt-get update - $SUDO apt-get install -y libgsl-dev - - - name: Install arcticpy build dependencies - shell: bash - run: | - set -euo pipefail - # --no-build-isolation reads NOTHING from arcticpy's build-system - # requires, so these must be present before the build starts. - python -m pip install --upgrade pip setuptools wheel - python -m pip install numpy cython - - - name: Install arcticpy runtime dependencies - shell: bash - run: | - set -euo pipefail - # --no-deps suppresses these, but arcticpy/read_noise.py imports both at - # import time (`from scipy.optimize import curve_fit`, `import - # matplotlib as mpl`) and __init__.py imports read_noise. So `import - # arcticpy` fails without them — which is why the verify step below, - # and every downstream `import autocti`, needs them present. - # - # The CTI stack installs scipy and matplotlib anyway as ordinary - # dependencies; naming them here is what makes this step verifiable on - # its own rather than only inside an already-built stack. - python -m pip install scipy matplotlib - - - name: Build and install arcticpy ${{ inputs.version }} - shell: bash - run: | - set -euo pipefail - python -m pip install "arcticpy==${{ inputs.version }}" \ - --no-build-isolation --no-deps - - - name: Verify arcticpy imports + # The recipe itself lives in install_arcticpy.sh beside this file, NOT in + # these steps. That is deliberate: `heart/smoke.py` prepares the same + # arcticpy for the LOCAL smoke runner, and a composite action cannot be + # invoked from Python. A shell script is the one shape both consumers can + # execute, so the recipe and the `2.6` pin stay in exactly one place — + # which is the whole point of #170 (it replaced four divergent copies). + # + # `github.action_path` is the directory this action was downloaded into, so + # this resolves for a consumer that has no checkout of PyAutoHeart — the + # property that made a composite action the right shape to begin with. + - name: Build and install arcticpy id: verify shell: bash - run: | - set -euo pipefail - # Assert here rather than letting a broken build surface much later as a - # confusing `import autocti` failure in an unrelated job. - # - # NOTE: arcticpy exposes no __version__ attribute — the obvious - # `import arcticpy; print(arcticpy.__version__)` raises AttributeError - # even on a perfectly good install. The distribution metadata is the - # supported way to ask. - VERSION="$(python -c 'import arcticpy; from importlib.metadata import version; print(version("arcticpy"))')" - echo "arcticpy $VERSION imported successfully" - echo "version=$VERSION" >> "$GITHUB_OUTPUT" + env: + ARCTICPY_VERSION: ${{ inputs.version }} + ARCTICPY_INSTALL_GSL: ${{ inputs.install-gsl }} + ARCTICPY_SUDO: ${{ inputs.sudo }} + run: bash "${{ github.action_path }}/install_arcticpy.sh" diff --git a/.github/actions/install-arcticpy/install_arcticpy.sh b/.github/actions/install-arcticpy/install_arcticpy.sh new file mode 100755 index 0000000..bc82bc3 --- /dev/null +++ b/.github/actions/install-arcticpy/install_arcticpy.sh @@ -0,0 +1,153 @@ +#!/usr/bin/env bash +# +# install_arcticpy.sh — THE canonical arcticpy install for the whole organism. +# +# This file is the single home of the recipe and of the `arcticpy==2.6` pin. +# Two consumers execute these exact bytes: +# +# * CI — `.github/actions/install-arcticpy/action.yml` (the composite action +# every CTI repo's workflow references) runs it via +# `${{ github.action_path }}/install_arcticpy.sh`. +# * Local — `heart/smoke.py` runs it directly out of the Heart checkout when a +# workspace's `smoke:` entry sets `arcticpy: true`. +# +# It is a shell script rather than only composite-action steps precisely so the +# local runner can share it: a composite action cannot be invoked from Python, +# and a second copy of the recipe is the failure this whole arrangement exists +# to prevent (PyAutoHeart#170 replaced four divergent copies with one). +# +# WHY THIS IS FIDDLY +# +# arcticpy is deliberately NOT a pip dependency of autocti: +# +# * its PyPI distribution is a source-only C++ sdist — it needs libgsl-dev +# headers and a toolchain to build, and there is no wheel; +# * its own requirements downgrade numpy below 2.0, which breaks a modern +# PyAuto stack. +# +# So it is built with --no-build-isolation (reusing the numpy already present +# instead of resolving its own) and --no-deps (so it cannot drag numpy back +# down). Each flag costs something that has to be paid back explicitly: +# +# --no-build-isolation pip does NOT create an isolated build environment and +# does NOT read pyproject's build-system.requires, so +# every BUILD dependency must already be installed. +# arcticpy declares none of them. +# --no-deps pip installs no RUNTIME dependencies either, so the +# packages arcticpy imports at import time must also be +# installed by hand. +# +# Both sets are installed below, and the script ends by proving the result +# actually imports. +# +# Verified 2026-08-24 in a clean container against a bare venv: each missing +# build dependency failed the build naming the next one. Without setuptools the +# build dies at `BackendUnavailable: Cannot import 'setuptools.build_meta'` — +# and Python 3.12+ venvs no longer ship setuptools by default, which is why the +# recipe that omitted it was a real hazard rather than a style nit. +# +# This script is intentionally SELF-CONTAINED: it upgrades pip/setuptools/wheel +# itself rather than relying on a caller having done so in a preceding step. +# +# ENVIRONMENT CONTRACT (all optional; defaults match the previous action) +# +# ARCTICPY_VERSION arcticpy version to build. Default 2.6. +# THE single pin for the organism — bump it HERE. +# ARCTICPY_INSTALL_GSL "true"/"false". Install libgsl-dev via apt. +# Default true. `false` means "GSL is already present", +# and the script PROVES that rather than assuming it. +# ARCTICPY_SUDO "true"/"false". Use sudo for the apt leg. Default +# true (GitHub-hosted runners); set false in a +# container already running as root. +# ARCTICPY_GSL_PREFIXES Space-separated include prefixes searched when the +# apt leg is skipped. Default covers Debian/Ubuntu, +# /usr/local and Homebrew-on-Apple-Silicon. Override it +# for the no-root workaround where GSL headers are +# extracted somewhere unprivileged (PyAutoCTI/AGENTS.md +# section arcticpy). +# PYTHON Interpreter to install into. Default `python`, which +# is correct both on a runner and inside an activated +# venv; heart/smoke.py passes its isolated venv's +# interpreter explicitly. +# +# On success, prints the installed version and — when $GITHUB_OUTPUT is set — +# writes `version=` to it, so the action's `version` output still works. + +set -euo pipefail + +ARCTICPY_VERSION="${ARCTICPY_VERSION:-2.6}" +ARCTICPY_INSTALL_GSL="${ARCTICPY_INSTALL_GSL:-true}" +ARCTICPY_SUDO="${ARCTICPY_SUDO:-true}" +PYTHON="${PYTHON:-python}" + +if [ "$ARCTICPY_INSTALL_GSL" = "true" ]; then + echo "==> Installing GSL headers" + SUDO="" + if [ "$ARCTICPY_SUDO" = "true" ]; then SUDO="sudo"; fi + $SUDO apt-get update + $SUDO apt-get install -y libgsl-dev +else + # The caller says GSL is already there. Check it, because the alternative is + # a compiler error hundreds of lines into the build naming a header, which + # reads as "arcticpy is broken" rather than "you are missing a system + # package". This is the path heart/smoke.py takes: a local dev command must + # not mutate system packages (and apt-get does not exist on macOS at all), + # so it declines to install GSL but must still fail legibly without it. + echo "==> Skipping GSL install (ARCTICPY_INSTALL_GSL=false); checking headers" + # Test each prefix independently. `ls a b c` is NOT the way to ask this: + # it exits non-zero when ANY operand is missing, so on a machine with GSL + # in exactly one of these three prefixes -- i.e. every machine that has it + # -- the check would report it absent. + GSL_PREFIXES="${ARCTICPY_GSL_PREFIXES:-/usr/include /usr/local/include /opt/homebrew/include}" + GSL_FOUND="false" + for prefix in $GSL_PREFIXES; do + if [ -f "$prefix/gsl/gsl_version.h" ]; then + echo " found GSL headers in $prefix" + GSL_FOUND="true" + break + fi + done + if [ "$GSL_FOUND" != "true" ]; then + echo "ERROR: GSL headers not found, and this invocation was told not to install them." >&2 + echo " arcticpy is a C++ sdist and cannot build without them. Install with:" >&2 + echo " Debian/Ubuntu: sudo apt-get install -y libgsl-dev" >&2 + echo " macOS: brew install gsl" >&2 + echo " Then re-run. (Searched: $GSL_PREFIXES — override with ARCTICPY_GSL_PREFIXES.)" >&2 + exit 1 + fi +fi + +echo "==> Installing arcticpy build dependencies" +# --no-build-isolation reads NOTHING from arcticpy's build-system requires, so +# these must be present before the build starts. +"$PYTHON" -m pip install --upgrade pip setuptools wheel +"$PYTHON" -m pip install numpy cython + +echo "==> Installing arcticpy runtime dependencies" +# --no-deps suppresses these, but arcticpy/read_noise.py imports both at import +# time (`from scipy.optimize import curve_fit`, `import matplotlib as mpl`) and +# __init__.py imports read_noise. So `import arcticpy` fails without them — +# which is why the verify step below, and every downstream `import autocti`, +# needs them present. +# +# The CTI stack installs scipy and matplotlib anyway as ordinary dependencies; +# naming them here is what makes this script verifiable on its own rather than +# only inside an already-built stack. +"$PYTHON" -m pip install scipy matplotlib + +echo "==> Building and installing arcticpy ${ARCTICPY_VERSION}" +"$PYTHON" -m pip install "arcticpy==${ARCTICPY_VERSION}" \ + --no-build-isolation --no-deps + +echo "==> Verifying arcticpy imports" +# Assert here rather than letting a broken build surface much later as a +# confusing `import autocti` failure in an unrelated job. +# +# NOTE: arcticpy exposes no __version__ attribute — the obvious +# `import arcticpy; print(arcticpy.__version__)` raises AttributeError even on a +# perfectly good install. The distribution metadata is the supported way to ask. +VERSION="$("$PYTHON" -c 'import arcticpy; from importlib.metadata import version; print(version("arcticpy"))')" +echo "arcticpy $VERSION imported successfully" +if [ -n "${GITHUB_OUTPUT:-}" ]; then + echo "version=$VERSION" >> "$GITHUB_OUTPUT" +fi diff --git a/.github/workflows/arcticpy-action.yml b/.github/workflows/arcticpy-action.yml index 4d684e1..f7c78ed 100644 --- a/.github/workflows/arcticpy-action.yml +++ b/.github/workflows/arcticpy-action.yml @@ -50,7 +50,12 @@ jobs: id: arctic uses: ./.github/actions/install-arcticpy with: - version: ${{ inputs.version || '2.6' }} + # Pass the dispatch input straight through. It used to fall back to a + # literal '2.6', which was a second copy of the pin: bumping the pin + # would not have changed what this self-test built, so the test would + # have gone on proving the OLD version worked. Empty means "use the + # action's own default", i.e. the one pin in install_arcticpy.sh. + version: ${{ inputs.version }} - name: Report the version the action installed run: echo "action reported arcticpy ${{ steps.arctic.outputs.version }}" diff --git a/config/repos.yaml b/config/repos.yaml index 7914017..1fdc634 100644 --- a/config/repos.yaml +++ b/config/repos.yaml @@ -142,6 +142,7 @@ smoke: PyAutoArray: autoarray PyAutoGalaxy: autogalaxy PyAutoLens: autolens + PyAutoCTI: autocti workspaces: autofit: directory: autofit_workspace @@ -161,6 +162,26 @@ smoke: howtolens: directory: HowToLens chain: [PyAutoNerves, PyAutoFit, PyAutoArray, PyAutoGalaxy, PyAutoLens] + # The CTI chain does NOT include autogalaxy/autolens — autolens sits on + # autogalaxy, autocti does not. It matches what both repos' CI callers + # declare. + # + # `arcticpy: true` mirrors the input those same CI callers pass to Heart's + # reusable smoke-tests.yml. `import autocti` hard-requires arcticpy, which is + # not a pip dependency (source-only C++ sdist, needs libgsl-dev, and its own + # requirements downgrade numpy below 2.0). CI gets it from the + # install-arcticpy composite action; a composite action cannot be invoked + # from smoke.py, so the local runner executes the same underlying script, + # `.github/actions/install-arcticpy/install_arcticpy.sh`. One recipe, one + # pin, both consumers — see PyAutoHeart#170/#172. + autocti: + directory: autocti_workspace + chain: [PyAutoNerves, PyAutoFit, PyAutoArray, PyAutoCTI] + arcticpy: true + autocti_test: + directory: autocti_workspace_test + chain: [PyAutoNerves, PyAutoFit, PyAutoArray, PyAutoCTI] + arcticpy: true # Excluded — not polled. Listed here for documentation only. excluded: diff --git a/heart/smoke.py b/heart/smoke.py index dd1ccad..718270d 100644 --- a/heart/smoke.py +++ b/heart/smoke.py @@ -19,6 +19,7 @@ import json import os import platform +import re import shlex import shutil import subprocess @@ -43,6 +44,22 @@ class WorkspaceSpec: key: str directory: str chain: tuple[str, ...] + #: Build arcticpy into this workspace's environment before running its + #: install epilogue. Mirrors the ``arcticpy: true`` input the CTI repos' + #: CI callers already pass to the reusable ``smoke-tests.yml`` — declared + #: rather than inferred from ``PyAutoCTI in chain``, so the local runner + #: and CI are configured by the same explicit statement. + arcticpy: bool = False + + +#: The one place the arcticpy recipe lives, shared with the CI composite action +#: that wraps it (``.github/actions/install-arcticpy/action.yml``). A composite +#: action cannot be invoked from Python, so the *script* is what the two +#: consumers have in common; duplicating the recipe here in Python would +#: re-create the divergence PyAutoHeart#170 removed. +ARCTICPY_INSTALLER = ( + HEART_HOME / ".github" / "actions" / "install-arcticpy" / "install_arcticpy.sh" +) def load_smoke_config( @@ -57,7 +74,12 @@ def load_smoke_config( cfg = yaml.safe_load(Path(config_path).read_text()) or {} block = cfg["smoke"] workspaces = { - key: WorkspaceSpec(key, spec["directory"], tuple(spec["chain"])) + key: WorkspaceSpec( + key, + spec["directory"], + tuple(spec["chain"]), + bool(spec.get("arcticpy", False)), + ) for key, spec in block["workspaces"].items() } return workspaces, dict(block["import_names"]) @@ -119,6 +141,13 @@ def environment_fingerprint( for path in watched if path.is_file() } + if spec.arcticpy and ARCTICPY_INSTALLER.is_file(): + # Keyed by a fixed label rather than a path relative to organism_root: + # the installer lives in the Heart checkout, which is not required to + # sit under the organism root at all. Hashing it means editing the + # recipe -- or bumping the arcticpy pin -- invalidates the cached + # environment, instead of silently reusing one built from the old one. + files["PyAutoHeart:install_arcticpy.sh"] = _sha256(ARCTICPY_INSTALLER) return { "schema": FINGERPRINT_SCHEMA, "workspace": spec.key, @@ -234,6 +263,36 @@ def _optional_local_targets(organism_root: Path, chain: Iterable[str]) -> list[s return targets +def _install_arcticpy(python: Path, env: dict[str, str]) -> None: + """Build arcticpy into ``python``'s environment via the shared recipe. + + Runs the very script the CI composite action runs, so there is exactly one + recipe, and one pin, in the organism. The GSL leg is disabled: a local + ``pyauto-heart smoke`` must not mutate system packages + (and ``apt-get`` does not exist on macOS), so the script proves the headers + are present and fails with the install line rather than reaching for sudo. + + ``PYTHON`` is passed explicitly rather than relying on the venv being first + on PATH -- this environment is prepared, not activated, and a bare + ``python`` resolving to the wrong interpreter would install arcticpy + somewhere the smoke run never looks. + """ + if not ARCTICPY_INSTALLER.is_file(): + raise SmokeEnvironmentError( + f"arcticpy installer missing: {ARCTICPY_INSTALLER}. This workspace " + "declares `arcticpy: true` in the `smoke:` block of " + "config/repos.yaml, which requires the shared recipe." + ) + _run( + ["bash", str(ARCTICPY_INSTALLER)], + env={ + **env, + "PYTHON": str(python), + "ARCTICPY_INSTALL_GSL": "false", + }, + ) + + def _install_environment( environment: Path, organism_root: Path, @@ -248,6 +307,9 @@ def _install_environment( ) _run([python, "-m", "pip", "install", "pyyaml"], env=env) + if spec.arcticpy: + _install_arcticpy(python, env) + workspace = organism_root / spec.directory installer = workspace / ".github" / "scripts" / "smoke_install.sh" if installer.is_file(): @@ -282,6 +344,49 @@ def _install_environment( ) +#: `pip check` lines that a CTI environment ALWAYS produces, and must. +#: +#: arcticpy declares ``numpy~=1.21`` but is installed with ``--no-deps`` on +#: purpose: honouring that requirement downgrades numpy below 2.0 and breaks the +#: rest of the PyAuto stack. The dependency metadata therefore stays permanently +#: unsatisfied, and ``pip check`` reports it every single time: +#: +#: arcticpy 2.6 has requirement numpy~=1.21, but you have numpy 2.5.2. +#: +#: Without this exception the preflight kills every CTI environment immediately +#: after building it, which would make `arcticpy: true` useless. The exception is +#: deliberately narrow -- one package, one requirement -- so a real conflict, in +#: arcticpy or anything else, still fails the preflight. +_ARCTICPY_EXPECTED_CONFLICT = re.compile( + r"^arcticpy \S+ has requirement numpy\S+, but you have numpy \S+\.?$" +) + + +def _pip_check(python: Path, env: Mapping[str, str], spec: WorkspaceSpec) -> None: + """Run ``pip check``, tolerating only the CTI stack's designed-in conflict.""" + # Goes through _run (not subprocess directly) so it stays consistent with + # every other command here -- same env handling, same mockability. + try: + _run([python, "-m", "pip", "check"], env=env, capture_output=True) + return + except subprocess.CalledProcessError as exc: + output = (exc.stdout or "") + (exc.stderr or "") + + lines = [line.strip() for line in output.splitlines() if line.strip()] + if spec.arcticpy: + lines = [ + line + for line in lines + if not _ARCTICPY_EXPECTED_CONFLICT.match(line) + and line != "No broken requirements found." + ] + if lines: + raise SmokeEnvironmentError( + "pip check reported broken requirements in " + f"{python}:\n " + "\n ".join(lines) + ) + + def _preflight( environment: Path, organism_root: Path, @@ -307,7 +412,7 @@ def _preflight( raise SmokeEnvironmentError( f"interpreter leak: expected {python}, subprocess used {executable}" ) - _run([python, "-m", "pip", "check"], env=env, capture_output=True) + _pip_check(python, env, spec) expected = { IMPORT_NAMES[repo]: str((organism_root / repo).resolve()) diff --git a/tests/test_repo_config.py b/tests/test_repo_config.py index 20ab649..cfd44a1 100644 --- a/tests/test_repo_config.py +++ b/tests/test_repo_config.py @@ -99,3 +99,56 @@ def test_required_workflows_block(config): assert "Navigator Check" in rw["workspaces"] for workflows in rw.values(): assert not any("url" in w.lower() for w in workflows) + + +def test_smoke_block_covers_every_cti_workspace(config): + """Both CTI smoke surfaces must be runnable through the local runner. + + Until PyAutoHeart#172 the `smoke:` block had no autocti entry at all and + `import_names` had no PyAutoCTI, so neither CTI suite could be exercised + without pushing and waiting for CI. + """ + smoke = config["smoke"] + assert smoke["import_names"]["PyAutoCTI"] == "autocti" + + workspaces = smoke["workspaces"] + for key, directory in ( + ("autocti", "autocti_workspace"), + ("autocti_test", "autocti_workspace_test"), + ): + spec = workspaces[key] + assert spec["directory"] == directory + # autocti does NOT sit on autogalaxy/autolens — autolens does. This + # mirrors what both repos' CI callers declare. + assert spec["chain"] == [ + "PyAutoNerves", + "PyAutoFit", + "PyAutoArray", + "PyAutoCTI", + ] + # `import autocti` hard-requires arcticpy, which is not a pip + # dependency; the flag mirrors the `arcticpy: true` input those same + # CI callers pass to the reusable smoke-tests.yml. + assert spec["arcticpy"] is True + + +def test_only_cti_workspaces_request_arcticpy(config): + """Every other workspace must be untouched by the arcticpy leg.""" + for key, spec in config["smoke"]["workspaces"].items(): + if key.startswith("autocti"): + continue + assert "arcticpy" not in spec, f"{key} should not request arcticpy" + + +def test_every_smoke_chain_repo_has_an_import_name(config): + """A chain repo missing from import_names is invisible to the preflight. + + The preflight proves each chain library imports from local source; it skips + any repo absent from the map, so an omission silently weakens the check + rather than failing it. That is how PyAutoCTI went unnoticed. + """ + smoke = config["smoke"] + known = set(smoke["import_names"]) + for key, spec in smoke["workspaces"].items(): + missing = [repo for repo in spec["chain"] if repo not in known] + assert not missing, f"{key} chain has no import name for {missing}" diff --git a/tests/test_smoke.py b/tests/test_smoke.py index fd113c5..b3ac4cf 100644 --- a/tests/test_smoke.py +++ b/tests/test_smoke.py @@ -393,3 +393,274 @@ def test_shell_wrapper_uses_explicit_smoke_python_before_module_import(): assert "--python) next_is_python=1" in body assert 'exec env PYTHONPATH="$HEART_HOME" "$runner_python" -m heart.smoke' in body + + +# --------------------------------------------------------------------------- +# arcticpy leg (PyAutoHeart#172) +# +# `import autocti` hard-requires arcticpy, which is not a pip dependency. CI gets +# it from the install-arcticpy composite action; a composite action cannot be +# invoked from Python, so the local runner executes the same underlying shell +# script. These tests pin the three properties that keep the two consumers from +# drifting apart, which is what PyAutoHeart#170 exists to prevent. +# --------------------------------------------------------------------------- + + +def test_arcticpy_defaults_off_and_is_read_from_config(tmp_path): + config = tmp_path / "repos.yaml" + config.write_text( + "smoke:\n" + " import_names: {LibraryB: libraryb}\n" + " workspaces:\n" + " plain:\n" + " directory: plain_workspace\n" + " chain: [LibraryB]\n" + " ctilike:\n" + " directory: ctilike_workspace\n" + " chain: [LibraryB]\n" + " arcticpy: true\n" + ) + workspaces, _ = smoke.load_smoke_config(config) + + # Absent means False, so every non-CTI workspace is untouched by this + # feature — the same default the CI input carries. + assert workspaces["plain"].arcticpy is False + assert workspaces["ctilike"].arcticpy is True + + +def test_fingerprint_tracks_the_shared_arcticpy_recipe(tmp_path, monkeypatch): + """Editing the recipe (or bumping the pin) must invalidate the cache. + + Without this the local runner would keep reusing an environment built from + the previous recipe, which is exactly the silent-divergence failure the + single-owner arrangement is meant to make impossible. + """ + installer = tmp_path / "install_arcticpy.sh" + installer.write_text("#!/usr/bin/env bash\nset -e\n") + monkeypatch.setattr(smoke, "ARCTICPY_INSTALLER", installer) + + identity = {"executable": "/fake/python", "version": "3.12.8"} + off = smoke.WorkspaceSpec("demo", "demo_workspace", ("LibraryB",)) + on = smoke.WorkspaceSpec("demo", "demo_workspace", ("LibraryB",), True) + root = make_tree(tmp_path, off) + + def digest(spec): + return smoke.fingerprint_digest( + smoke.environment_fingerprint(root, spec, identity) + ) + + # A workspace that does not use arcticpy is not affected by the recipe. + off_before = digest(off) + on_before = digest(on) + assert off_before != on_before + + installer.write_text("#!/usr/bin/env bash\nset -e\n# bumped pin\n") + assert digest(off) == off_before + assert digest(on) != on_before + + +def test_install_runs_the_shared_recipe_before_the_epilogue(tmp_path, monkeypatch): + """Order matters: CI installs arcticpy *then* runs the workspace epilogue. + + Also asserts the two things the local invocation must get right — an + explicit interpreter (the environment is prepared, not activated, so a bare + `python` could install arcticpy where the smoke run never looks) and + GSL install disabled (a local dev command must not mutate system packages). + """ + installer = tmp_path / "install_arcticpy.sh" + installer.write_text("#!/usr/bin/env bash\nset -e\n") + monkeypatch.setattr(smoke, "ARCTICPY_INSTALLER", installer) + + spec = smoke.WorkspaceSpec("demo", "demo_workspace", ("LibraryB",), True) + root = make_tree(tmp_path, spec) + environment = tmp_path / "env" + smoke._environment_bin(environment).mkdir(parents=True) + smoke._environment_python(environment).write_text("") + + calls: list[tuple[list[str], dict]] = [] + + def fake_run(command, **kwargs): + calls.append(([str(part) for part in command], kwargs.get("env") or {})) + return completed() + + monkeypatch.setattr(smoke, "_run", fake_run) + smoke._install_environment( + environment, root, spec, {"version": "3.12.8"} + ) + + commands = [command for command, _ in calls] + arctic = next(i for i, c in enumerate(commands) if str(installer) in c) + epilogue = next( + i for i, c in enumerate(commands) if "smoke_install.sh" in " ".join(c) + ) + assert arctic < epilogue + + _, env = calls[arctic] + assert env["PYTHON"] == str(smoke._environment_python(environment)) + assert env["ARCTICPY_INSTALL_GSL"] == "false" + + +def test_install_reports_a_missing_recipe_rather_than_skipping_it( + tmp_path, monkeypatch +): + monkeypatch.setattr(smoke, "ARCTICPY_INSTALLER", tmp_path / "absent.sh") + spec = smoke.WorkspaceSpec("demo", "demo_workspace", ("LibraryB",), True) + root = make_tree(tmp_path, spec) + environment = tmp_path / "env" + smoke._environment_bin(environment).mkdir(parents=True) + smoke._environment_python(environment).write_text("") + monkeypatch.setattr(smoke, "_run", lambda *a, **k: completed()) + + # Silently carrying on would surface much later as a confusing + # `import autocti` failure inside an unrelated script. + with pytest.raises(smoke.SmokeEnvironmentError, match="arcticpy installer missing"): + smoke._install_environment(environment, root, spec, {"version": "3.12.8"}) + + +def test_shipped_arcticpy_installer_exists_and_is_the_only_pin(): + """The declared recipe must actually be on disk where smoke.py looks. + + `arcticpy: true` in config/repos.yaml is a promise that this file exists; + a rename in .github/ would otherwise only surface when someone ran a CTI + smoke suite locally. + """ + assert smoke.ARCTICPY_INSTALLER.is_file() + body = smoke.ARCTICPY_INSTALLER.read_text() + assert 'ARCTICPY_VERSION="${ARCTICPY_VERSION:-' in body + + +def _run_installer(tmp_path, prefixes: str) -> subprocess.CompletedProcess[str]: + """Run the shared recipe far enough to exercise the GSL probe. + + PYTHON points at a stub that fails, so the run stops at the first pip call. + Everything after the probe is pip work this test has no interest in. + """ + stub = tmp_path / "python-stub" + stub.write_text("#!/usr/bin/env bash\nexit 77\n") + stub.chmod(0o755) + return subprocess.run( + ["bash", str(smoke.ARCTICPY_INSTALLER)], + env={ + "PATH": os.environ["PATH"], + "ARCTICPY_INSTALL_GSL": "false", + "ARCTICPY_GSL_PREFIXES": prefixes, + "PYTHON": str(stub), + }, + capture_output=True, + text=True, + ) + + +def test_recipe_gsl_probe_accepts_headers_in_any_one_prefix(tmp_path): + """The probe must test each prefix independently. + + The first cut used `ls a b c`, which exits non-zero when ANY operand is + missing — so on every machine that actually has GSL (in exactly one prefix) + it reported the headers absent and refused to build. Caught by running the + thing rather than reading it. + """ + present = tmp_path / "present" + (present / "gsl").mkdir(parents=True) + (present / "gsl" / "gsl_version.h").write_text("") + absent = tmp_path / "absent" + absent.mkdir() + + result = _run_installer(tmp_path, f"{absent} {present}") + + assert "found GSL headers" in result.stdout + assert "GSL headers not found" not in result.stderr + + +def test_recipe_refuses_without_gsl_and_says_how_to_fix_it(tmp_path): + """A missing system package must not surface as a compiler error. + + Without this the failure is hundreds of lines into a C++ build naming a + header, which reads as "arcticpy is broken" rather than "install libgsl-dev". + """ + absent = tmp_path / "absent" + absent.mkdir() + + result = _run_installer(tmp_path, str(absent)) + + assert result.returncode == 1 + assert "GSL headers not found" in result.stderr + assert "apt-get install -y libgsl-dev" in result.stderr + assert "brew install gsl" in result.stderr + + +def test_recipe_never_reaches_for_sudo_when_gsl_install_is_disabled(tmp_path): + """The local path must not mutate system packages, ever.""" + present = tmp_path / "present" + (present / "gsl").mkdir(parents=True) + (present / "gsl" / "gsl_version.h").write_text("") + + result = _run_installer(tmp_path, str(present)) + + combined = result.stdout + result.stderr + assert "apt-get update" not in combined + assert "Installing GSL headers" not in combined + + +# --------------------------------------------------------------------------- +# pip check vs the CTI stack's designed-in conflict +# +# arcticpy declares numpy~=1.21 and is installed with --no-deps on purpose: +# honouring that requirement downgrades numpy below 2.0 and breaks the rest of +# the stack. So `pip check` reports it in EVERY CTI environment, forever. Found +# by running the local runner end to end, not by reading it — the environment +# built correctly and was then destroyed by its own preflight. +# --------------------------------------------------------------------------- + + +def _pip_check_failing(monkeypatch, output: str): + def fake_run(command, **kwargs): + if "check" in [str(part) for part in command]: + raise subprocess.CalledProcessError( + 1, command, output=output, stderr="" + ) + return completed() + + monkeypatch.setattr(smoke, "_run", fake_run) + + +ARCTICPY_CONFLICT = "arcticpy 2.6 has requirement numpy~=1.21, but you have numpy 2.5.2." + + +def test_pip_check_tolerates_the_arcticpy_numpy_conflict(tmp_path, monkeypatch): + _pip_check_failing(monkeypatch, ARCTICPY_CONFLICT + "\n") + spec = smoke.WorkspaceSpec("demo", "demo_workspace", ("LibraryB",), True) + + smoke._pip_check(Path("/fake/python"), {}, spec) # must not raise + + +def test_pip_check_still_fails_on_any_other_conflict(tmp_path, monkeypatch): + """The exception is one package and one requirement wide, not a blanket skip.""" + _pip_check_failing( + monkeypatch, + ARCTICPY_CONFLICT + "\nautofit 1.0 has requirement autoarray>=9, but you have autoarray 1.\n", + ) + spec = smoke.WorkspaceSpec("demo", "demo_workspace", ("LibraryB",), True) + + with pytest.raises(smoke.SmokeEnvironmentError) as excinfo: + smoke._pip_check(Path("/fake/python"), {}, spec) + message = str(excinfo.value) + assert "autofit 1.0 has requirement" in message + assert "arcticpy" not in message + + +def test_pip_check_does_not_tolerate_arcticpy_for_a_non_arcticpy_workspace( + tmp_path, monkeypatch +): + """A workspace that never asked for arcticpy has no business carrying it.""" + _pip_check_failing(monkeypatch, ARCTICPY_CONFLICT + "\n") + spec = smoke.WorkspaceSpec("demo", "demo_workspace", ("LibraryB",)) + + with pytest.raises(smoke.SmokeEnvironmentError, match="arcticpy"): + smoke._pip_check(Path("/fake/python"), {}, spec) + + +def test_pip_check_passes_through_when_clean(tmp_path, monkeypatch): + monkeypatch.setattr(smoke, "_run", lambda *a, **k: completed()) + spec = smoke.WorkspaceSpec("demo", "demo_workspace", ("LibraryB",), True) + + smoke._pip_check(Path("/fake/python"), {}, spec) # must not raise From 732ec63f2d4367cf31eec0a4da31e5374dbef4fd Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 24 Aug 2026 21:34:25 +0000 Subject: [PATCH 2/2] smoke: keep satellite repo names out of organ code (tenant firewall) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI's `repos_sync.py --check --only "tenant firewall (organ code)"` failed the pytest job — after all 602 tests passed — on three instance facts the previous commit introduced. The firewall keeps satellite repo names out of PyAutoBrain / PyAutoHeart / PyAutoHands so an adopting fork has a small, declared set of files to rewrite. tests/test_smoke.py's own fixture comment says exactly this, and I added the names anyway. * heart/smoke.py and install_arcticpy.sh are UNLISTED, where any instance fact is drift. Both mentions were prose in comments and carried no weight: "inferred from ``PyAutoCTI in chain``" -> "inferred from the chain's contents", and a pointer to the CTI library's AGENTS.md by name -> by role. * tests/test_repo_config.py is allowlisted, but for {PyAutoCTI, autocti_workspace, autocti_workspace_test} only; the new chain assertion added PyAutoArray and PyAutoFit. Rather than grow the entry — the list's own comment says never to do that casually — the assertion now goes through `import_names`, comparing package names (autonerves/autofit/autoarray/ autocti) instead of repo-name literals. It says the same thing and additionally proves every chain repo resolves in the map, so it is a slightly better test than the one it replaces. Verified by running the real check, not by inspection: with PyAutoHeart symlinked under a root using the casing the checker expects (it silently SKIPS an organ directory it cannot find, so a lowercase checkout gives a vacuous OK), it reports OK — and still reports the mismatch when a fact is deliberately planted. 602 tests still pass. Refs #172 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014wiN7R1yaeGj6k1Pa1FEq4 --- .../install-arcticpy/install_arcticpy.sh | 4 ++-- heart/smoke.py | 2 +- tests/test_repo_config.py | 18 ++++++++++-------- 3 files changed, 13 insertions(+), 11 deletions(-) diff --git a/.github/actions/install-arcticpy/install_arcticpy.sh b/.github/actions/install-arcticpy/install_arcticpy.sh index bc82bc3..d29451d 100755 --- a/.github/actions/install-arcticpy/install_arcticpy.sh +++ b/.github/actions/install-arcticpy/install_arcticpy.sh @@ -63,8 +63,8 @@ # apt leg is skipped. Default covers Debian/Ubuntu, # /usr/local and Homebrew-on-Apple-Silicon. Override it # for the no-root workaround where GSL headers are -# extracted somewhere unprivileged (PyAutoCTI/AGENTS.md -# section arcticpy). +# extracted somewhere unprivileged (see the CTI +# library's AGENTS.md, section arcticpy). # PYTHON Interpreter to install into. Default `python`, which # is correct both on a runner and inside an activated # venv; heart/smoke.py passes its isolated venv's diff --git a/heart/smoke.py b/heart/smoke.py index 718270d..dacefdd 100644 --- a/heart/smoke.py +++ b/heart/smoke.py @@ -47,7 +47,7 @@ class WorkspaceSpec: #: Build arcticpy into this workspace's environment before running its #: install epilogue. Mirrors the ``arcticpy: true`` input the CTI repos' #: CI callers already pass to the reusable ``smoke-tests.yml`` — declared - #: rather than inferred from ``PyAutoCTI in chain``, so the local runner + #: rather than inferred from the chain's contents, so the local runner #: and CI are configured by the same explicit statement. arcticpy: bool = False diff --git a/tests/test_repo_config.py b/tests/test_repo_config.py index cfd44a1..96aadcc 100644 --- a/tests/test_repo_config.py +++ b/tests/test_repo_config.py @@ -118,14 +118,16 @@ def test_smoke_block_covers_every_cti_workspace(config): ): spec = workspaces[key] assert spec["directory"] == directory - # autocti does NOT sit on autogalaxy/autolens — autolens does. This - # mirrors what both repos' CI callers declare. - assert spec["chain"] == [ - "PyAutoNerves", - "PyAutoFit", - "PyAutoArray", - "PyAutoCTI", - ] + # Asserted through import_names rather than as repo-name literals: the + # tenant firewall (PyAutoMind/scripts/repos_sync.py) treats a satellite + # repo name in organ code as an instance fact, and this says the same + # thing while additionally proving every chain repo resolves in the map. + chain_packages = [smoke["import_names"][repo] for repo in spec["chain"]] + assert chain_packages == ["autonerves", "autofit", "autoarray", "autocti"] + # The CTI stack does NOT sit on the galaxy/lens libraries — the lens + # stack does. This mirrors what both repos' CI callers declare. + assert "autogalaxy" not in chain_packages + assert "autolens" not in chain_packages # `import autocti` hard-requires arcticpy, which is not a pip # dependency; the flag mirrors the `arcticpy: true` input those same # CI callers pass to the reusable smoke-tests.yml.