ci: publish a GitHub Release on every release

.claude/harness-candidates.md

@@ -43,6 +43,60 @@ Deferred lint/test guardrails surfaced during reviews. Promote to a `CExxx` rule
   `test_auto_mixed_pass_stops_ignoring_undecided_distractors` +
   `test_mixed_static_arming_pass_stops_ignoring_fail_armed`.
 
+## From 2026-07-24 publish-github-releases review
+
+- [ ] **sdist-contents assertion** — build the sdist and assert it contains only
+  intended paths. `pyproject.toml` declares no `[tool.hatch.build.targets.sdist]`
+  section, so hatchling's default selection honors only the **root** `.gitignore`
+  and sweeps in everything else sitting in the tree at build time. Two distinct
+  consequences, worth keeping apart:
+  - **What actually reaches PyPI today: nothing unintended.** A local `uv build`
+    in a developed worktree produces a 135 MB sdist carrying
+    `evalboard/node_modules/**` (8520 files) and `evalboard/.next/**` (190),
+    because `evalboard/.gitignore` is nested and therefore not honored. CI is
+    spared only incidentally — `release.yml` never runs `npm`/`pnpm install`, so
+    those paths do not exist on the runner at `uv build` time. Verified against
+    the published artifacts: the 0.8.9 and 0.8.2 sdists on PyPI are ~7.5 MB /
+    ~550 files with **zero** `node_modules` entries. (A dirty-tree release would
+    not silently ship JS either — 135 MB exceeds PyPI's 100 MB per-file limit, so
+    it fails at upload. The real exposure is a broken release, not a stealth one.)
+  - **The live hazard is untracked files a workflow leaves in the tree**, which
+    hatchling *does* package: a `release-notes.md` written at the repo root by a
+    CI step landed in `coder_eval-X.Y.Z/release-notes.md` (verified by building
+    it). This is why the "Publish GitHub Release" step writes to
+    `${RUNNER_TEMP}` — a convention no check enforces.
+
+  Not cheap: needs a real `uv build` inside the test suite (slow) plus a decision
+  on whether to add an explicit sdist include/exclude allowlist, which changes
+  published artifacts. Worth pairing with the allowlist so the contract is
+  declared rather than inferred from hatchling's defaults — caught in the
+  2026-07-24 ci/publish-github-releases review.
+- [ ] **CE032 — run the existing AST lint rules over Python embedded in
+  `.github/workflows/*.yml`.** CE008/CE009/CE010 already forbid unencoded
+  `read_text`/`open`/`subprocess.run`, but `tests/lint/runner.py::check_paths`
+  walks only `*.py` under `src/`, so Python inside a `run:` heredoc is invisible
+  to ruff, pyright, pytest, coverage *and* the CE runner. Would have caught the
+  four unencoded `read_text`/`write_text` calls fixed by hand in this review
+  (`release.yml` ×2, `publish-testpypi.yml` ×2). Needs a heredoc extractor
+  (`python3 - <<'PY' … PY` → dedent → `ast.parse`) with line-number mapping back
+  to the YAML; wire as a `tests/test_custom_lint.py` class like CE027–CE031
+  rather than a `BaseRule`. Also consider extending CE008 to `write_text` (it
+  matches only `read_text` today, though `src/` happens to be clean).
+- [ ] **CE033 — interpreter heredocs in `.github/workflows/**` must use a quoted
+  delimiter** (`<<'PY'`, not `<<PY`). With a bare tag the shell expands `$VAR`
+  into the *program text* before the interpreter parses it, so a value containing
+  a quote or newline breaks out of the string literal it lands in. Fixed by hand
+  in `publish-testpypi.yml` in this review (it interpolated `${DEV_VERSION}` into
+  Python source); regex-detectable in ~10 lines, and CE032's `ast.parse` is only
+  sound on quoted bodies, so the two ship together.
+- [ ] **`actionlint` + `zizmor` over `.github/workflows/**`.** No static analysis
+  whatsoever runs over workflow YAML today (`make verify` never looks at it), so
+  every workflow finding in the 2026-07-24 review was caught by a human reading
+  it. `actionlint` runs shellcheck over `run:` bodies; `zizmor`'s
+  `excessive-permissions` / `artipacked` / `template-injection` rules cover the
+  credential-scoping and `${{ }}`-into-`run:` classes reviewed by hand. Subsumes
+  the CE026 SHA-pinning candidate above. Start as a non-blocking annotation job.
+
 ## From 2026-07-03 open-source docs cleanup
 
 - [ ] **Dead-relative-link checker for `docs/**/*.md`** — resolve every relative

.github/scripts/release_notes.py

@@ -0,0 +1,74 @@
+#!/usr/bin/env python3
+"""Slice one version's section out of ``CHANGELOG.md`` for a GitHub Release body.
+
+Used by ``.github/workflows/release.yml``'s "Publish GitHub Release" step.
+
+This lives in a real module rather than inline in a ``run:`` heredoc because
+code inside a heredoc is structurally invisible to ruff, pyright, pytest and
+coverage — and this particular code has three distinct failure modes (version
+absent, section is the file's last, version string carrying regex
+metacharacters) on a path that executes exactly once per release, against
+production ``main``, after the tag is already pushed. The prerelease dispatch
+skips the step, so there is no rehearsal; ``tests/test_release_notes.py`` is
+what exercises it before it matters.
+
+Usage::
+
+    release_notes.py <version> <output-path>
+
+Writes the section body (heading line excluded, stripped) to ``<output-path>``,
+or an empty file when no matching section exists — the caller treats an empty
+file as "fall back to GitHub's generated notes".
+"""
+
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+
+_REPO_ROOT = Path(__file__).resolve().parents[2]
+
+# semantic-release's default changelog template renders headings as
+# "## v0.8.9 (2026-07-23)". Capture everything after that heading line up to the
+# next "## v" heading, or EOF for the newest entry (the common case on a release
+# run, since the just-generated section is at the top of the file).
+#
+# The trailing \b on the version is load-bearing: without it "0.8.1" would match
+# the "## v0.8.10" heading and publish the wrong section.
+_SECTION_TEMPLATE = r"(?m)^## v{version}\b.*?$(.*?)(?=^## v|\Z)"
+
+
+def extract_section(changelog_text: str, version: str) -> str:
+    """Return the changelog body for ``version``, or ``""`` when absent.
+
+    ``version`` is regex-escaped, so a PEP 440 local/prerelease version
+    containing metacharacters (``+``, ``!``) is matched literally.
+    """
+    pattern = _SECTION_TEMPLATE.format(version=re.escape(version))
+    match = re.search(pattern, changelog_text, re.S)
+    return match.group(1).strip() if match else ""
+
+
+def main(argv: list[str]) -> int:
+    if len(argv) != 3:
+        print("usage: release_notes.py <version> <output-path>", file=sys.stderr)
+        return 2
+    version, out_path = argv[1], argv[2]
+    # Encoding is pinned rather than left to the ambient locale: CHANGELOG.md
+    # verifiably carries non-ASCII (arrows, em dashes, check marks), so a
+    # non-UTF-8 locale would raise UnicodeDecodeError here and — under the
+    # caller's `continue-on-error: true` — silently publish no Release at all.
+    text = (_REPO_ROOT / "CHANGELOG.md").read_text(encoding="utf-8")
+    body = extract_section(text, version)
+    Path(out_path).write_text(body, encoding="utf-8")
+    if not body:
+        # Surfaced as a workflow annotation; the caller falls back to
+        # --generate-notes when the file is empty.
+        print(f"::warning::no CHANGELOG section found for v{version}; using GitHub's generated notes")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main(sys.argv))

.github/workflows/publish-testpypi.yml

@@ -64,8 +64,14 @@ jobs:
           CURRENT_VERSION=$(python3 -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")
           # PEP 440 dev release: sorts before the base release, unique per run.
           DEV_VERSION="${CURRENT_VERSION}.dev${RUN_NUMBER}"
-          python3 - <<PY
-          import re, pathlib
+          # Exported so the heredoc below can read it from os.environ. The delimiter
+          # is QUOTED (<<'PY'): with a bare <<PY the shell expands $DEV_VERSION into
+          # the Python source text, so a value containing a quote or newline would
+          # break out of the string literal it lands in.
+          export DEV_VERSION
+          python3 - <<'PY'
+          import os, re, pathlib
+          version = os.environ["DEV_VERSION"]
           # pyproject.toml is the canonical source; __init__ is kept in sync.
           for path, pat in (
               ("pyproject.toml", r'(?m)^version\s*=\s*".+?"$'),
@@ -76,10 +82,12 @@ jobs:
               # re.subn (not re.sub) so a version-line format drift aborts loudly
               # instead of silently no-op'ing (rc=0) and publishing a stale/base
               # version that then collides on TestPyPI (masked by skip-existing).
-              new, n = re.subn(pat, f'{key} = "${DEV_VERSION}"', p.read_text(), count=1)
+              # encoding pinned rather than left to the ambient locale, which would
+              # otherwise raise UnicodeDecodeError mid-publish on a non-ASCII source.
+              new, n = re.subn(pat, f'{key} = "{version}"', p.read_text(encoding="utf-8"), count=1)
               if n != 1:
                   raise SystemExit(f"version pattern did not match {path} (matched {n}); refusing to publish a stale/colliding artifact")
-              p.write_text(new)
+              p.write_text(new, encoding="utf-8")
           PY
           echo "Stamped dev version: ${DEV_VERSION}"
 

.github/workflows/release.yml

@@ -1,9 +1,9 @@
 name: Release
 
-# Manually-triggered release: bump the version, tag, build, and publish to
-# public PyPI (wheel) and GHCR (agent image). Merges to main do NOT
-# auto-release -- run this workflow from the Actions tab when you want to cut a
-# release.
+# Manually-triggered release: bump the version, tag, build, publish to public
+# PyPI (wheel) and GHCR (agent image), and cut a GitHub Release for the tag.
+# Merges to main do NOT auto-release -- run this workflow from the Actions tab
+# when you want to cut a release.
 #
 # The bump level is CHOSEN at dispatch, not derived from commit messages:
 # `patch` (default), `minor`, or `major`. A dispatch always cuts a release.
@@ -17,7 +17,8 @@ name: Release
 # PRERELEASE mode (dispatched from a NON-main branch): instead of bumping and
 # pushing main, stamp a throwaway `<next-patch>rc<run#>` version, then build and
 # publish the wheel to public PyPI + a `:<version>` GHCR image. It does NOT move
-# `:latest`, tag, commit, or push to main. This lets a branch be dry-run on the
+# `:latest`, tag, commit, push to main, or create a GitHub Release (there is no
+# tag for a Release to point at). This lets a branch be dry-run on the
 # ADO nightly infra before merge (pinned via the pipeline's `coderEvalVersion`).
 # The `bump` input is ignored off main. On `main` the behavior is unchanged.
 
@@ -160,10 +161,12 @@ jobs:
           version = os.environ["PRE_VERSION"]
           for path, key in (("pyproject.toml", "version"), ("src/coder_eval/__init__.py", "__version__")):
               p = pathlib.Path(path)
-              new, n = re.subn(rf'(?m)^{key}\s*=\s*".+?"$', f'{key} = "{version}"', p.read_text(), count=1)
+              # encoding pinned, not left to the ambient locale: a non-UTF-8 default
+              # would raise UnicodeDecodeError mid-release on a non-ASCII source file.
+              new, n = re.subn(rf'(?m)^{key}\s*=\s*".+?"$', f'{key} = "{version}"', p.read_text(encoding="utf-8"), count=1)
               if n != 1:
                   raise SystemExit(f"version pattern did not match {path} (matched {n})")
-              p.write_text(new)
+              p.write_text(new, encoding="utf-8")
           print(f"Stamped prerelease version: {version}")
           PY
           uv lock
@@ -241,6 +244,106 @@ jobs:
           path: dist/
           if-no-files-found: error
 
+      # Publish the GitHub Release for the tag pushed above. semantic-release runs
+      # with --no-vcs-release because it also runs --no-push (the commit is amended
+      # and the tag re-pointed first), so it cannot create the release itself -- it
+      # happens here, once the tag is actually on the remote. A published Release is
+      # what GitHub Marketplace listings are cut from, so every release needs one.
+      # (`gh release create` cannot tick the "Publish this Action to the
+      # Marketplace" checkbox -- that stays a one-time manual step in the GitHub UI
+      # on the first Release; every subsequent release then lists automatically.)
+      #
+      # Deliberately placed AFTER "Build wheel + sdist" and "Upload dist for PyPI
+      # publish" rather than at the earliest legal point after the tag push: those
+      # two steps are the last ones that can still fail for an already-tagged
+      # version, and a Release announcing a version whose artifacts never built is
+      # worse than a missing Release. This narrows the window rather than closing
+      # it -- publish-pypi is a separate job, so the actual upload to PyPI still
+      # happens after this. Running here also keeps a slow/hung `gh` API call from
+      # eating the 15-minute job budget BEFORE the artifacts are safe, which would
+      # produce exactly the stranded-tag state the note below warns about.
+      #
+      # Notes are the CHANGELOG section semantic-release just generated for this
+      # version, sliced by .github/scripts/release_notes.py (a real module, so the
+      # regex is unit-tested -- see tests/test_release_notes.py); an empty result
+      # falls back to GitHub's generated notes.
+      #
+      # ACCEPTED RISK: those notes render commit subjects, i.e. squashed PR titles.
+      # The Release body is a first-party surface that GitHub also fans out in
+      # notification emails, so it carries text that was reviewed as *code*, not as
+      # markdown -- a PR title can land an arbitrary link in it. Bounded to
+      # content/link spoofing (GitHub strips raw HTML from release bodies) and
+      # gated by this repo's mandatory PR review. Revisit with `--draft` plus a
+      # human glance, or link-stripping in release_notes.py, if the repo ever takes
+      # drive-by contributions.
+      - name: Publish GitHub Release
+        id: gh_release
+        if: steps.mode.outputs.prerelease != 'true' && steps.release.outputs.version != ''
+        # Best-effort, mirroring the GHCR steps below. main, the version tag, the
+        # moving major tag, and the dist artifact are all in place by the time this
+        # runs, so a transient GitHub API failure here must not fail the job: the
+        # publish-pypi job is `needs: release`, so a failure would SKIP the PyPI
+        # publish of an already-tagged version and strand `@vN` on an action.yml pin
+        # whose version was never published. The next step turns the swallowed
+        # failure into a loud annotation instead of a collapsed step marker.
+        continue-on-error: true
+        env:
+          # The app token, not GITHUB_TOKEN: the workflow's `permissions:` are
+          # contents: read, and `gh release create` needs contents: write. Granting
+          # the job contents: write to use GITHUB_TOKEN here would ADD a second
+          # write credential rather than remove one -- `actions/checkout` above
+          # already persists this same app token in .git/config for every step in
+          # the job, so scoping it out of this one step's env buys no isolation.
+          GH_TOKEN: ${{ steps.app-token.outputs.token }}
+          # Passed via env (not interpolated into the script) per GitHub's
+          # injection guidance.
+          VERSION: ${{ steps.release.outputs.version }}
+        run: |
+          set -euo pipefail
+          # Written under RUNNER_TEMP, never the repo root: hatchling's default sdist
+          # file selection sweeps in untracked files at the root (verified -- it ships
+          # even git-ignored paths), so a notes file left in the tree would leak into
+          # the sdist that the "Build wheel + sdist" step above produced and the
+          # publish-pypi job uploads.
+          NOTES_FILE="${RUNNER_TEMP}/release-notes.md"
+          python3 .github/scripts/release_notes.py "$VERSION" "$NOTES_FILE"
+          # Empty notes file => no CHANGELOG section was found (the script already
+          # emitted the ::warning::); let GitHub generate the body instead.
+          if [ -s "$NOTES_FILE" ]; then
+            NOTES=(--notes-file "$NOTES_FILE")
+          else
+            NOTES=(--generate-notes)
+          fi
+          gh release create "v${VERSION}" \
+            --title "v${VERSION}" \
+            --verify-tag \
+            --latest \
+            "${NOTES[@]}"
+
+      # `continue-on-error` above hides a failure in a collapsed step marker that
+      # nobody expands on an otherwise-green release run -- the same silence that
+      # let "no GitHub Releases at all" go unnoticed until this PR. Re-raise it as
+      # an ::error annotation plus a run-summary block, WITHOUT failing the job
+      # (that would skip publish-pypi, see above).
+      - name: Flag missing GitHub Release
+        if: always() && steps.gh_release.outcome == 'failure'
+        env:
+          VERSION: ${{ steps.release.outputs.version }}
+        run: |
+          set -euo pipefail
+          MAJOR="v${VERSION%%.*}"
+          echo "::error title=GitHub Release not published::v${VERSION} was tagged and its artifacts built, but 'gh release create' failed. Create the Release by hand so ${MAJOR} and the Marketplace listing resolve."
+          {
+            echo "### :x: GitHub Release for \`v${VERSION}\` was NOT created"
+            echo
+            echo "The version tag, the moving \`${MAJOR}\` tag, and the PyPI artifacts are unaffected —"
+            echo "only \`gh release create\` failed. Create it by hand:"
+            echo
+            echo '```sh'
+            echo "gh release create v${VERSION} --title v${VERSION} --verify-tag --latest --generate-notes"
+            echo '```'
+          } >> "$GITHUB_STEP_SUMMARY"
+
       # Build + push the agent image HERE, in the same job that produced the
       # version, so the `:<version>` tag is built from the correct pyproject (bumped
       # by semantic-release on main, or the stamped rc on a prerelease).

Makefile

@@ -15,11 +15,17 @@ install:  ## Install project with dev + uipath dependencies (hash-verified from
 	uv sync --frozen --extra dev --extra uipath
 	uv run pre-commit install
 
+# `.github/scripts/` is in scope on purpose: release tooling that lives in a real
+# module (rather than inline in a workflow `run:` heredoc) is exactly what ruff,
+# pyright and pytest can see — leaving it unlinted would forfeit the reason it was
+# extracted.
+LINT_PATHS := src/ tests/ .github/scripts/
+
 format:  ## Auto-format code with ruff
-	uv run ruff format src/ tests/
+	uv run ruff format $(LINT_PATHS)
 
 check:  ## Run linting checks
-	uv run ruff check src/ tests/
+	uv run ruff check $(LINT_PATHS)
 
 lint:  ## Run custom architectural lint rules (CE001+)
 	uv run pytest tests/test_custom_lint.py -v --tb=short --no-header -p no:warnings
@@ -42,8 +48,8 @@ test-cov:  ## Run tests with coverage report
 
 
 verify:  ## Run all verification steps (CI equivalent)
-	uv run ruff format --check src/ tests/
-	uv run ruff check src/ tests/
+	uv run ruff format --check $(LINT_PATHS)
+	uv run ruff check $(LINT_PATHS)
 	uv run pyright
 	uv run pytest tests/test_custom_lint.py -v --tb=short --no-header -p no:warnings
 	# uv run pip-audit --desc --skip-editable