Skip to content

Clear the reproducible backlog, and test every format deterministically - #37

Open
okcodes wants to merge 7 commits into
mainfrom
backlog-sweep
Open

okcodes wants to merge 7 commits into
mainfrom
backlog-sweep

Conversation

@okcodes

@okcodes okcodes commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

Four backlog items, each reproduced before being fixed, plus the deterministic
coverage that makes checking a format by hand unnecessary.

The e2e gap came first

Four of the ten supported layouts had no end-to-end coverage at all —
pyproject.toml, uv.lock, Directory.Build.props and .targets were proven
only by unit tests, which is why the newest formats were the ones being checked
by hand.

One table now drives two tests: a stable bump and a pre-release, each asserting
every declared file moved and that check agrees in both directions. Adding a
format is a row. Verified by breaking one writer and watching both go red.

1 · npm workspaces

Reproduced: a project declaring a member's package.json with the shared
package-lock.json was told its files disagreed — when they agreed exactly.
vump was reading the root entry.

This looked like it needed a path mapping, which is the inference the entry
refused to guess at. It doesn't: npm records the name inside each
packages entry, so a member is selected by name like Cargo and uv.
package.json joins the manifests read for that name.

The root's two records belong to the project only when the lock's own name is
one the project declares — that is what stops a member rewriting the root above
it.

2 · Inputs on the check action

Two gaps, treated differently:

  • project input — added. A repository declaring several projects and
    verifying a bare version could not use the action at all. A tag needs
    nothing: it selects its own project through tag_pattern, which is why this
    went unnoticed.
  • allow-nested input — deliberately refused, and the description now says
    so. A nested vump.toml answers about the wrong project, and check is the
    command whose verdict gets believed. Adding it would weigh the convenience of
    a layout [[project]] exists to prevent.

Found while in the run step: inputs were expanded straight into bash, where
${{ }} is substituted before bash parses the line, so a version carrying
shell syntax would have executed. They now arrive through the environment.

3 · vump push

git push --follow-tags pushes every annotated tag reachable from HEAD.
Reproduced against a bare remote: it published other-project-v9.9.9, which
had nothing to do with the release.

vump push names one ref, derived from the version the files record and the
project's pattern — no stored state, and it cannot name a tag other than the
one that exists. It refuses when that tag is absent rather than pushing the
branch alone. The branch policy is checked here too, since a bump stopping at
tag was never checked against it.

The hint printed after tagging now says vump push.

4 · A Rust workspace sandbox — the entry was wrong

The entry claimed a crate in this tree needs excluding from the workspace, and
that a mistake breaks cargo build for vump itself. Neither holds: this
repository's Cargo.toml declares no [workspace], so a nested one is simply
separate. cargo metadata still reports one member with the sandbox in place.

What it was right about is that the shape worth showing had no worked example.
Two crates sharing one lock now show it, with --locked asserting cargo
accepts what vump wrote.

The manual pass, automated

./scripts/sandbox-smoke.sh bumps every sandbox and asks node, dotnet, uv and
cargo what version they build — the one thing the suite cannot answer.

Deliberately not in cargo test: that would put four toolchains and a
network on three runners to test somebody else's tool, and the formats are
already covered deterministically above. Missing toolchains skip; everything
reverts on the way out.

Backlog

Before After
Ideas 7 3
Lines 257 199

The three left are open product questions with no reproduction: declarative
formats, commit-message bumps, changelog generation.

🤖 Generated with Claude Code

okcodes and others added 6 commits October 21, 2026 23:53
Four of the ten supported layouts had no end-to-end coverage at all —
pyproject.toml, uv.lock, Directory.Build.props and .targets were proven only
by unit tests, so the formats added most recently were the ones being checked
by hand.

One table now drives two tests: a stable bump and a pre-release, each asserting
that every declared file moved and that `check` agrees in both directions.
Adding a format is a row. Breaking one writer turns both red, which is how the
table was verified.

The pre-release case is not redundant with the stable one. It writes a version
that is not three numbers, which is where a format that cannot hold one shows
up at all.

Co-Authored-By: Claude Opus 5 <[email protected]>
A project declaring a member's package.json alongside the shared
package-lock.json was told its files disagreed, naming the member's version
against the root's. They did not disagree: the lock's entry for that member
held exactly what the manifest held. vump was reading the root.

The lock keys entries by path, which is why this looked like it needed a
mapping from a declared manifest path to a `packages` key — the inference the
backlog entry refused to guess at. It does not: each entry records the name
inside it, so a member is selected by the name its manifest declares, as in
every other lock. package.json joins the manifests read for that name.

The root's two records, the top-level version and the "" entry, now belong to
the project only when the lock's own name is one the project declares. That is
what stops a member rewriting the root above it, and it leaves a lock with no
`packages` map — or one declared without a manifest to name it — reading the
root as before.

Co-Authored-By: Claude Opus 5 <[email protected]>
…nto bash

A repository declaring several projects and verifying a bare version rather
than a tag could not use this action: vump refuses to guess which project, and
there was no input to tell it. A tag needs nothing — it selects its own project
through that project's tag_pattern — which is why this went unnoticed.

There is deliberately no allow-nested input, and the description now says so.
A vump.toml nested below another answers about the wrong project, and `check`
is the command whose verdict gets believed; the action failing there is the
refusal working. Adding the input would weigh the convenience of a layout
[[project]] exists to prevent.

The step also expanded ${{ inputs.* }} straight into a shell line, where an
expression is substituted before bash parses it — a version input carrying
shell syntax would have run as part of the step. The values now arrive through
the environment, as the download step already does, and `set -euo pipefail`
means a failure in the cd cannot leave the check running somewhere else.

Co-Authored-By: Claude Opus 5 <[email protected]>
Finishing by hand meant `git push --follow-tags`, which carries every
annotated tag reachable from HEAD. In a repository tagging several projects
independently that publishes whichever other project's tag was sitting
unpushed — the release workflow on the far end cannot tell the difference.

`vump push` names one ref. It stores nothing: the tag comes from the version
the files already record and the project's own pattern, which is the same
derivation the bump used to create it, so it cannot name a tag other than the
one that exists. Where that tag does not exist it refuses rather than pushing
the branch alone, which would look like the job was done.

The branch policy is checked here too. A bump stopping at `tag` was never
checked against it, because nothing had left the machine; this is the step
that would put the tag on the remote.

The hint printed after tagging now names this command instead of the two git
commands it replaces. Reading one agreed version from the tracked files was
inline in `bump`, and is now shared rather than written twice.

Co-Authored-By: Claude Opus 5 <[email protected]>
…red it

The entry said a crate in this tree would need excluding from the workspace,
and that a mistake there breaks `cargo build` for the tool itself. Neither
holds: this repository's Cargo.toml declares no [workspace], so it is its own
single member and a nested workspace is simply a separate one. `cargo
metadata` still reports one member with the sandbox in place, and `publish =
false` makes the packaged-contents question moot too.

What it was right about is that the shape worth showing had no worked example.
The repository demonstrates a single crate; a workspace — one lock entry per
member, matched by the name each manifest declares — existed only in test
fixtures. Two crates released together now show it, and `--locked` is the
assertion that vump wrote what cargo expects rather than something cargo would
rewrite.

Co-Authored-By: Claude Opus 5 <[email protected]>
The suite proves vump writes the bytes it meant to. Whether those bytes reach
a running program is a question only dotnet, uv, npm and cargo can answer, and
it was being answered by hand once per format.

Deliberately a script rather than a test. Putting it in `cargo test` would put
four toolchains and a network on three CI runners, making every job slower and
flakier in order to test somebody else's tool — and the formats it exercises
are already covered deterministically by the table in tests/cli.rs.

A missing toolchain is skipped rather than failed, every project is reverted on
the way out including after a failure, and a bump that fails is recorded as a
failure rather than skipping the assertion that followed it — which the first
version of this script did, reporting success while every step had failed.

Co-Authored-By: Claude Opus 5 <[email protected]>
The script reverts sandbox/ when it finishes, which is a destructive git
command in a real repository rather than in a temporary one. Anything sitting
there unsaved went with it.

Co-Authored-By: Claude Opus 5 <[email protected]>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant