diff --git a/AGENTS.md b/AGENTS.md index b325f69..179892a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -183,14 +183,16 @@ bin/idstack-manifest-merge --section --payload # Canonical manifest wri bin/idstack-slugify "" # Derive the for .idstack/exports/ ``` -Tests — all eight run in CI on every push and PR (`.github/workflows/test.yml`, +Tests — all ten run in CI on every push and PR (`.github/workflows/test.yml`, ubuntu on Python 3.9 + 3.12, macOS on 3.12): ```bash ./test/smoke-test.sh # Install, SKILL.md freshness, frontmatter, version agreement, # canonical section names, namespacing, resolve-snippet lockstep -./test/integration-test.sh # End-to-end; proves the suite leaves the working tree untouched +./test/integration-test.sh # Behavioral tests across bin/; proves the suite leaves the tree untouched ./test/test-setup.sh # ./setup behavior: flags, scope, legacy cleanup, failure handling +./test/test-doctor.sh # bin/idstack-doctor PROBLEM/WARNING branches +./test/test-status.sh # bin/idstack-status, including the --readiness export gate ./test/test-manifest-merge.sh # bin/idstack-manifest-merge unit tests ./test/test-version-classifier.sh # bin/lib/version-classify.sh unit tests ./test/test-plugin-status.sh # bin/lib/plugin-status.sh unit tests @@ -198,6 +200,10 @@ ubuntu on Python 3.9 + 3.12, macOS on 3.12): ./test/mutation-test.sh # Reintroduces each fixed defect, asserts its guarding test fails ``` +`test/test-helper.sh` is sourced by every suite rather than run: it owns the +shared `PASS`/`FAIL`/`TOTAL` counters and the `check()` assertion. A suite must +not define its own counters — smoke-test fails on one. + Python 3.9 is the oldest interpreter in the field (macOS system python3) and the leg that catches modern-only syntax reaching the preamble's embedded scripts. diff --git a/CLAUDE.md b/CLAUDE.md index 5e5f34a..479197e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,13 +27,16 @@ bin/idstack-manifest-merge --section --payload # Canonical manifest wr bin/idstack-slugify "" # Derive the used for .idstack/exports/ ``` -Tests (all eight run in CI on every push and PR — see `.github/workflows/test.yml`): +Tests (all ten run in CI on every push and PR — see `.github/workflows/test.yml`): ```bash -./test/smoke-test.sh # 371 assertions: install, SKILL.md freshness, frontmatter, version agreement, +./test/smoke-test.sh # 393 assertions: install, SKILL.md freshness, frontmatter, version agreement, # canonical section names, /idstack: namespacing, resolve-snippet lockstep, bash -n -./test/integration-test.sh # End-to-end run; proves the suite leaves the working tree untouched +./test/integration-test.sh # 48 behavioral tests across the bin/ scripts; also proves the suite + # leaves the working tree untouched ./test/test-setup.sh # 17 behavioral tests for ./setup (flags, scope, legacy cleanup, failure handling) +./test/test-doctor.sh # 13 behavioral tests for bin/idstack-doctor's PROBLEM/WARNING branches +./test/test-status.sh # 22 tests for bin/idstack-status, incl. the --readiness export gate ./test/test-manifest-merge.sh # bin/idstack-manifest-merge unit tests ./test/test-version-classifier.sh # bin/lib/version-classify.sh unit tests ./test/test-plugin-status.sh # bin/lib/plugin-status.sh unit tests @@ -41,6 +44,8 @@ Tests (all eight run in CI on every push and PR — see `.github/workflows/test. ./test/mutation-test.sh # Reintroduces each fixed defect and asserts its guarding test fails ``` +`test/test-helper.sh` is not a suite — it is sourced by all of them and owns the shared `PASS`/`FAIL`/`TOTAL` counters and the `check()` assertion. Do not add a local counter block to a suite; smoke-test fails on one, and a mutation proves that guard works. + CI matrix: ubuntu (Python 3.9 + 3.12) and macOS (3.12). 3.9 is the leg that catches modern-only Python syntax reaching the preamble's embedded scripts — it is what macOS ships. `mutation-test.sh` runs once, pinned to 3.9. No build step for users. No dependencies beyond bash (python3 recommended for full features). Skills are plain Markdown files. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1d66c92..42d52fd 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -129,13 +129,15 @@ bin/idstack-gen-skills --dry-run # Generated files up to date? Run this first. ./test/smoke-test.sh # Verify installation across all targets ``` -Eight suites run in CI (`.github/workflows/test.yml`) on every push and pull request. Run whichever ones your change touches locally; CI runs them all: +Ten suites run in CI (`.github/workflows/test.yml`) on every push and pull request. Run whichever ones your change touches locally; CI runs them all: | Suite | Covers | |-------|--------| | `test/smoke-test.sh` | Install, `SKILL.md` freshness, YAML frontmatter, version agreement across `VERSION` / `plugin.json` / `CHANGELOG.md`, canonical manifest section names, `/idstack:` namespacing, resolve-snippet lockstep, schema migrations, `bash -n` on every script | -| `test/integration-test.sh` | End-to-end run; asserts the suite leaves your working tree untouched | +| `test/integration-test.sh` | Behavioral tests across the `bin/` scripts; also asserts the suite leaves your working tree untouched | | `test/test-setup.sh` | `./setup` behavior — flag parsing, scope selection, all three legacy-cleanup shapes, failure handling. Runs against a repo copy with a fake `$HOME` and a stub `claude` | +| `test/test-doctor.sh` | `bin/idstack-doctor` — every PROBLEM/WARNING branch and the exit contract. Runs against a repo copy with a pinned `PATH`, so "claude not found" means genuinely not found | +| `test/test-status.sh` | `bin/idstack-status`, including the `--readiness` export gate. Each threshold is probed at its own boundary with the other two held passing | | `test/test-manifest-merge.sh` | `bin/idstack-manifest-merge` | | `test/test-version-classifier.sh` | `bin/lib/version-classify.sh` | | `test/test-plugin-status.sh` | `bin/lib/plugin-status.sh` | @@ -144,6 +146,22 @@ Eight suites run in CI (`.github/workflows/test.yml`) on every push and pull req The CI matrix is ubuntu on Python 3.9 and 3.12, plus macOS on 3.12 for BSD `grep`/`sed`/`awk` differences. `mutation-test.sh` runs once, pinned to 3.9. Do not skip hooks or push with a red suite. +### Writing a new assertion + +`test/test-helper.sh` is sourced by every suite and owns the counters and the assertion: + +```bash +. "$(dirname "$0")/test-helper.sh" + +check "description" "some-command" # must exit 0 +check "rejects a bad payload" "$MERGE --section bogus" 3 # must exit 3 +check "names the failing skill" "$STATUS --readiness" 0 "INCOMPLETE" +``` + +Arguments 3 and 4 are optional: an expected exit code, and an ERE the output must match. On failure the first five lines of output are printed. Do not add a private `PASS=0`/`check()` to a suite — smoke-test asserts none exist, and a mutation proves that guard works. Nine suites each had their own copy once and they drifted: two spelled it `assert`, and one swallowed failure output entirely, so CI reported a bare `FAIL` with nothing to act on. + +Two suites keep a differently-shaped wrapper because their assertion is not "run a command": `check_version` compares a version string to a classification, `check_listing` feeds a `claude plugin list` listing to `plugin_is_enabled`. Both are named so they cannot shadow `check`. + Then test manually: - Claude Code: `/idstack:your-skill` - Codex CLI: `$your-skill` @@ -159,7 +177,9 @@ Then test manually: ## Looking for something to work on? -[TODOS.md](TODOS.md) is the backlog — deferred work with the reasoning and dependencies attached. [ROADMAP.md](ROADMAP.md) is the user-facing view of the same thing. Read [DESIGN.md](DESIGN.md) before changing anything visual; it is the source of truth for the report stylesheet and the landing page. +Two places, on purpose. [TODOS.md](TODOS.md) is the long-lived feature backlog — deferred work with the reasoning and dependencies attached. [Open issues](https://github.com/savvides/idstack/issues) hold work that came out of a specific code review or PR triage, so it stays next to the evidence that produced it; TODOS.md links the current ones. [ROADMAP.md](ROADMAP.md) is the user-facing view. + +Read [DESIGN.md](DESIGN.md) before changing anything visual; it is the source of truth for the report stylesheet and the landing page. ## Questions or feedback? diff --git a/ROADMAP.md b/ROADMAP.md index 5b41793..aeca45c 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -4,6 +4,11 @@ What's coming next for idstack. Priorities are shaped by user feedback. [Tell us ## Just shipped +### Follow-up fixes to the v3.3.0.0 audit (v3.3.0.1–v3.3.0.4) +- **Skills stopped suggesting commands that don't exist.** Welcome-back and next-step messages named things like `/assessment-design`, which resolves in neither CLI. It landed inside the context-recovery message v3.3.0.0 had just repaired, so it was the first thing you saw once those messages started working again (v3.3.0.1). +- **Searching your learnings for a word with an apostrophe returned the wrong answer.** `Bloom's` or `learner's` made the search code a syntax error, which was swallowed, and the fallback that took over ignores the `--type` filter — so it answered with a record of the wrong type instead of failing. Those are ordinary search terms in this tool (v3.3.0.2). +- Smaller: a malformed payload can no longer corrupt the manifest, and on machines without python3 the search fallback's `--type` filter works at all (it had never matched anything idstack wrote). + ### Course memory, pipeline orchestration, and re-run detection fixed (v3.3.0.0) - **Welcome-back messages work on stock macOS.** The session-memory code embedded in every skill contained an f-string that is a SyntaxError on any Python below 3.12 — including the 3.9 macOS ships. It failed silently, so context recovery, quality-score trends, and next-step suggestions produced nothing. Fixed, and now exercised on 3.9 in CI so the class of bug can't ship again. - **`/idstack:pipeline` invokes its child skills correctly.** It was calling them by an unnamespaced name that never resolved in Claude Code. @@ -12,10 +17,11 @@ What's coming next for idstack. Priorities are shaped by user feedback. [Tell us - **Apostrophes no longer blank the dashboard**, imported courses get a proper next-step suggestion, and `bin/idstack-doctor` can no longer report a disabled install as healthy. - **Standalone runs persist.** `bin/idstack-migrate --init` creates a canonical manifest, so a skill run outside the pipeline has something to write into instead of silently discarding its results. -### Test infrastructure and CI (v3.3.0.0, for contributors) -- The suite had never run automatically. GitHub Actions now runs all eight suites on every push and pull request, across ubuntu (Python 3.9 + 3.12) and macOS. +### Test infrastructure and CI (v3.3.0.0–v3.3.0.4, for contributors) +- The suite had never run automatically. GitHub Actions runs every suite on each push and pull request, across ubuntu (Python 3.9 + 3.12) and macOS — eight suites at v3.3.0.0, ten today. - `./setup` — the primary deliverable — went from zero coverage to 17 behavioral tests. -- `test/mutation-test.sh` reintroduces each fixed defect and asserts its guarding test fails, which is how a test that only appeared to test something gets caught. +- `bin/idstack-doctor` and `bin/idstack-status --readiness` gained their first execution coverage in v3.3.0.4. Both are what a user reaches for when something has already gone wrong, and neither had any. +- `test/mutation-test.sh` reintroduces each fixed defect and asserts its guarding test fails, which is how a test that only appeared to test something gets caught. 18 mutations, all guarded. ### Install through the Claude Code plugin marketplace (v3.2.0.0) - `./setup` registers idstack as a Claude Code plugin marketplace and installs from there. Recent Claude Code versions stopped discovering plugins from the bare symlink older setups created, so `/idstack:` commands silently never appeared in the slash picker. If that happened to you, pull the latest and re-run `./setup`. diff --git a/TODOS.md b/TODOS.md index 9e1bbe3..89c2238 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,15 @@ # TODOS +> **Also check the open GitHub issues.** This file holds the long-lived feature +> backlog. Work that came out of a specific code review or PR triage is tracked +> as an issue instead, so it stays next to the evidence that produced it: +> +> - [#60](https://github.com/savvides/idstack/issues/60) — an interrupted `idstack-learnings-delete` can destroy the whole log (truncate-in-place) +> - [#61](https://github.com/savvides/idstack/issues/61) — decide whether `--cross-project` lets global learnings outrank local ones; the code and the comment above it disagree +> - [#62](https://github.com/savvides/idstack/issues/62) — remaining test-coverage gaps, one of which is a silent regression today +> +> The issue is authoritative for its own detail; don't copy it here. + ## ~~v2.4: Dual-Output Report Contract + Pipeline Aggregator~~ SHIPPED (v2.4.0.0) Shipped in v2.4.0.0 with Markdown reports under `.idstack/reports/` (historical — later releases replaced that path with self-contained HTML reports under diff --git a/templates/agent-context.md b/templates/agent-context.md index b325f69..179892a 100644 --- a/templates/agent-context.md +++ b/templates/agent-context.md @@ -183,14 +183,16 @@ bin/idstack-manifest-merge --section --payload # Canonical manifest wri bin/idstack-slugify "" # Derive the for .idstack/exports/ ``` -Tests — all eight run in CI on every push and PR (`.github/workflows/test.yml`, +Tests — all ten run in CI on every push and PR (`.github/workflows/test.yml`, ubuntu on Python 3.9 + 3.12, macOS on 3.12): ```bash ./test/smoke-test.sh # Install, SKILL.md freshness, frontmatter, version agreement, # canonical section names, namespacing, resolve-snippet lockstep -./test/integration-test.sh # End-to-end; proves the suite leaves the working tree untouched +./test/integration-test.sh # Behavioral tests across bin/; proves the suite leaves the tree untouched ./test/test-setup.sh # ./setup behavior: flags, scope, legacy cleanup, failure handling +./test/test-doctor.sh # bin/idstack-doctor PROBLEM/WARNING branches +./test/test-status.sh # bin/idstack-status, including the --readiness export gate ./test/test-manifest-merge.sh # bin/idstack-manifest-merge unit tests ./test/test-version-classifier.sh # bin/lib/version-classify.sh unit tests ./test/test-plugin-status.sh # bin/lib/plugin-status.sh unit tests @@ -198,6 +200,10 @@ ubuntu on Python 3.9 + 3.12, macOS on 3.12): ./test/mutation-test.sh # Reintroduces each fixed defect, asserts its guarding test fails ``` +`test/test-helper.sh` is sourced by every suite rather than run: it owns the +shared `PASS`/`FAIL`/`TOTAL` counters and the `check()` assertion. A suite must +not define its own counters — smoke-test fails on one. + Python 3.9 is the oldest interpreter in the field (macOS system python3) and the leg that catches modern-only syntax reaching the preamble's embedded scripts.