Skip to content

ci: release-grade install matrix, upgrade test and docs-as-tests - #67

Merged
danoszz merged 6 commits into
mainfrom
ci/release-verification
Oct 20, 2026
Merged

danoszz merged 6 commits into
mainfrom
ci/release-verification

Conversation

@danoszz

@danoszz danoszz commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

Release-grade verification for 1.8.0. Until now CI ran Penthera only from the repo, on Linux. This PR tests the packed tarball the way users install it, on Linux, macOS and Windows. It also tests an upgrade from 1.7.0 and every command shown in the docs.

Found and fixed along the way (separate commits):

  • Windows: npx penthera / penthera init was broken in any git repo. It exited 1 with Refusing to write outside the project: D:\…\reports\…. The cause: git rev-parse --show-toplevel prints D:/a/app with forward slashes, and that was compared against resolve()d backslash paths. Both git-root helpers (src/cli/detect.js, lib/hook/index.js) now return native paths (6cf41d6).
  • penthera --help was missing --rules-signers and --allow-unsigned-rules. Both are parsed and documented in docs/rule-packs.md (3399fc1).

No version bump, no tag.

What's new

.github/workflows/install-matrix.yml runs on PRs to main, pushes to main and workflow_dispatch. It uses permissions: contents: read and SHA-pinned actions (download-artifact v8.0.1 is the only new pin).

  • npm pack runs once (--ignore-scripts, since ci.yml already runs the tests), and the job checks the tarball's file list: required files present, no repo-only folders. The tarball is passed on as an artifact.
  • fresh-install: ubuntu/macos/windows × Node 22/24. The tarball is installed into an empty project (npm install <tgz> --ignore-scripts), and scripts/ci/smoke-installed.mjs runs npx penthera against a committed git copy of tests/fixtures/init/app (lockfile + .mcp.json + CLAUDE.md). Each check asserts the exit code and key output lines.
  • upgrade: on ubuntu/macos/windows, [email protected] is installed from npm in a temp HOME and runs a --memory scan plus ui --demo. Then the tarball is installed over it, and the job confirms the lockfile now resolves to file:. Old memory must verify (penthera memory verify), memory show must read it, and a new --memory scan must extend the same chain (6 → 9 events). ui --demo must start against the old ~/.penthera.
  • install matrix result: a single aggregate check for branch protection.

scripts/docs-commands.mjs + tests/docs-commands.test.js (runs in npm test, and in the matrix against the installed package on all three OSes).

scripts/ci/net-guard.cjs is preloaded via NODE_OPTIONS into every Penthera process in these checks. A TCP/TLS connection or DNS query past loopback fails like a real network error and is logged, and the check asserts the log is empty. So a command that quietly falls back after a failed download still fails.

Matrix results (run 39414300920)

fresh install (17 checks) docs commands upgrade from 1.7.0
ubuntu · node 22 ✅ 17/17 ✅ 140/140 ran ✅ 5/5 + 10/10
ubuntu · node 24 ✅ 17/17 (node 22 only)
macos · node 22 ✅ 17/17 ✅ 140/140 ran ✅ 5/5 + 10/10
macos · node 24 ✅ 17/17
windows · node 22 ✅ 17/17 ✅ 140/140 ran ✅ 5/5 + 10/10
windows · node 24 ✅ 17/17

Fresh-install checks: --version; --help; init --yes --dry-run (detects git, lockfile and agent config, plans the hook and workflow, and writes nothing, which is checked with a file snapshot); deps; deps --base HEAD (spawns git); agents; --repo .. Next, hook install, then a real git commit through the hook: a staged key is blocked and a clean commit passes. That is Git for Windows' sh running node.exe by absolute path. Then hook uninstall. ui --demo is started in the background and polled until / answers 401 without a token, then stopped, and the check confirms the port is closed (no orphaned grandchild under npx). rules verify runs on both bundled packs from the installed package, then rules list. Last comes the zero-network assertion.

Windows notes

  • Path handling: the init bug above. Everything else, including the hook's absolute node.exe path, worked.
  • Line endings: the Windows checkout gives the fixture CRLF (autocrlf), and the smoke log prints it. deps, agents, init and the hook all read CRLF lockfiles and agent config correctly.
  • Spawned git/npm: npx goes through npx.cmd with a shell. git is spawned by init, deps --base, the hook and the commit. The ui server is stopped with taskkill /T.
  • Cursor skill symlink: .cursor/skills/penthera → skills/penthera is a link only for people working on Penthera in Cursor. GitHub's Windows runner checks it out as a real link, but a default Git for Windows install gives a text file. This is documented in CONTRIBUTING ("Windows": Developer Mode + git clone -c core.symlinks=true). The npm package does not ship .cursor/, so users are unaffected. A Windows step records what the checkout got.

Docs-command coverage

334 commands in 40 files (fenced bash blocks, run: lines in yaml blocks, inline penthera … spans). Every Penthera command is parse-checked: the subcommand, the action (memory show, rules verify, …) and every flag must appear in the matching --help.

class n handling
runnable 140 ran on the fixture under the net guard, with substitutions (--base origin/main → HEAD, --repo ./my-app → .). Commands run per file in document order, so --cra-init cra.json makes --cra cra.json runnable. All pass
placeholder 111 flags only: <url>, $URL, interactive (bare penthera, init without --yes, agent --loop), long-running ui, localhost:3000 needing an app, inputs the fixture lacks (./my-pack, ../base-checkout), --machine, --sign-key
external 44 flags only: public URLs (32), penthera install (npm registry, 17 incl. inline), --suppliers
other 37 not Penthera: npx skills add, curl … | bash, /plugin, ssh-keygen, cp, jq, competitor CLIs in docs/compare/, …
planned 2 penthera ship / watch in docs/plan.md: planning docs (plan.md, vision-2.0.md, release-plan.md, research/, funding/) are reported, not failed

The only wrong command found was the help gap above. No README or doc text changed. node scripts/docs-commands.mjs --report out.md prints the full breakdown, and the matrix writes it to each job summary.

Network

No new job hits the public internet from Penthera. npm does contact the registry in two places: installing the tarball fetches its one runtime dependency (yaml), as any user install does, and the upgrade job installs [email protected]. Both are documented in the workflow header and CONTRIBUTING.

Checks

  • npm test green locally (910 passed, docs test adds about 10 s) and in CI on Node 22/24.
  • npm run audit:internal passes all gates locally (REPORT.md not committed).
  • CodeQL: 4 incomplete-escaping alerts on the new report writers, fixed in 30a0a57.
  • actionlint clean.

Follow-ups (not in this PR)

  • After 1.8.0 ships, move FROM_VERSION in install-matrix.yml to 1.8.0.
  • Consider making install matrix result a required check.
  • The repo's own unit suite still runs on Linux only. Running npm test on Windows/macOS is a separate, bigger change: some tests use /dev/null and POSIX shells.

🤖 Generated with Claude Code

danoszz and others added 3 commits October 20, 2026 14:15
Both flags are parsed and documented in docs/rule-packs.md but were missing
from `penthera --help`. Found by the new docs-commands check.

AI-Assisted: Claude Code with Claude Opus 5.5; found the gap with the
  docs-commands check and added the two help lines
Co-Authored-By: Claude Opus 5.5 <[email protected]>
…rom 1.7.0, and every documented command

- install-matrix.yml: npm pack once; install the tarball into an empty
  project on ubuntu/macos/windows x Node 22/24 and run npx penthera
  (--version, --help, init --yes --dry-run, deps, deps --base, agents,
  --repo, hook install + a real git commit through the hook, ui --demo
  answering 401, rules verify on the bundled packs).
- Upgrade job: [email protected] from npm writes memory and ui state, then
  the tarball must verify and extend the same chain.
- scripts/docs-commands.mjs: extracts every command in README, docs and
  SKILL.md, parse-checks subcommands and flags against --help and runs
  the offline-safe ones on a fixture. Runs in npm test and in the matrix
  against the installed package.
- scripts/ci/net-guard.cjs: every Penthera process in these checks fails
  on a connection or DNS query past loopback.

AI-Assisted: Claude Code with Claude Opus 5.5; designed and wrote the
  workflow, smoke runner, network guard and docs checker from the
  maintainer's release-verification brief
Co-Authored-By: Claude Opus 5.5 <[email protected]>
AI-Assisted: Claude Code with Claude Opus 5.5; wrote the notes for the new
  install matrix, upgrade job, docs-commands check and the Windows symlink
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Comment thread scripts/ci/smoke-installed.mjs Fixed
Comment thread scripts/docs-commands.mjs Fixed
Comment thread scripts/docs-commands.mjs Fixed
Comment thread scripts/docs-commands.mjs Fixed
danoszz and others added 3 commits October 20, 2026 14:20
…ject

On Windows, `git rev-parse --show-toplevel` prints D:/a/app with forward
slashes. The wizard compared resolve()d report paths (D:\a\app\reports)
against that root, so every write failed with "Refusing to write outside the
project" and `npx penthera` / `penthera init` exited 1 in any git repo.
Both git-root helpers now return native paths. Found by the Windows legs of
the new install matrix.

AI-Assisted: Claude Code with Claude Opus 5.5; traced the install-matrix
  failure to the git path format and normalized both helpers
Co-Authored-By: Claude Opus 5.5 <[email protected]>
CodeQL (js/incomplete-sanitization) on the new report writers.

AI-Assisted: Claude Code with Claude Opus 5.5; fixed the CodeQL alerts
Co-Authored-By: Claude Opus 5.5 <[email protected]>
AI-Assisted: Claude Code with Claude Opus 5.5; added the changelog line
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@danoszz
danoszz marked this pull request as ready for review October 20, 2026 11:24
@danoszz
danoszz merged commit 81f40e2 into main Oct 20, 2026
16 checks passed
@danoszz
danoszz deleted the ci/release-verification branch October 20, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants