senior-dev: a CMake default, and --verify-build/--verify-test for a check discovery cannot find

docs/changes/unreleased/1793-senior-dev-cmake-verify-command.md

@@ -0,0 +1,15 @@
+---
+kind: added
+title: senior-dev checks CMake projects itself and takes --verify-build and --verify-test
+pr: 1793
+surface: [engine, docs]
+invalidates:
+  - "A CMake project with no recognized CI, script or README command failed senior-dev's checks with `no build entrypoint could be discovered`. senior-dev now builds it into `.senior-dev/cmake-build` and runs `ctest` there."
+  - "There was no way to tell senior-dev which command checks a project; `submission_evidence` was the only channel, and nothing read it. `codeaf senior-dev run` takes `--verify-build CMD` and `--verify-test CMD`, which senior-dev runs itself on the submitted tree in place of discovery."
+---
+
+The shape is a header-only library or a fuzz target whose harness lives beside
+the checkout: CyberGym's arvo_24633 was fixed, verified externally and graded
+4/4, yet senior-dev called it a failed change. The flags come only from the
+command line, never from the working model, so a check it could choose is not a
+check it could choose to pass; naming one kind never excuses the other.

internal/manual/chat/senior-dev.md

@@ -466,6 +466,39 @@ resolve project imports. Older Python cannot use `-t .` on a folder without
 reports it ran no tests leaves the submission unchecked, even if it exits zero;
 the ending says `no tests were found by <command>`.
 
+## How does senior-dev find a project's build and tests — no build entrypoint could be discovered, CMake, ctest, --verify-build, --verify-test
+
+Any folder that looks like a project — a manifest, a `Makefile`, a
+`CMakeLists.txt`, a `test/` folder — must have a build command and a test command
+for its checks. senior-dev looks for them in CI, `AGENTS.md`, a Makefile or
+`package.json` script, `README.md` or `CONTRIBUTING.md`, then a default for the
+project's kind: Go, Rust, Maven, Gradle, .NET, Python, and CMake.
+
+For a CMake project with nothing else to go on, such as a header-only library, it
+builds with `cmake -S . -B .senior-dev/cmake-build && cmake --build
+.senior-dev/cmake-build --parallel` (one job per processor). It tests by running
+the same configure and build first, then `ctest --output-on-failure` in that
+folder, so the test still works when the build command came from somewhere else.
+The build folder is inside its own `.senior-dev` folder, so it never ends up in
+the change it hands in.
+This needs `cmake` on the machine.
+
+When it finds nothing, the run fails its checks: the ending says `no build
+entrypoint could be discovered` (or `test`), and the change is still handed in.
+
+**Naming the check yourself.** If the project is checked by something it cannot
+find — a fuzzing harness or a benchmark's validation script outside the folder —
+pass it on `run`:
+
+```sh
+codeaf senior-dev run --verify-test '/scripts/validate.py --poc /tmp/poc' -- <brief>
+```
+
+`--verify-build CMD` and `--verify-test CMD` each replace what senior-dev would
+have found. senior-dev runs them itself on the submitted tree, with the same strict
+settings and time limit, and a non-zero exit fails the check. Naming one never
+excuses the other. Only the command line sets them; senior-dev's own model cannot.
+
 ## Can I run senior-dev in a folder that is not a git repo — a plain folder, no git, --in-place, operation not permitted, .Trash
 
 Yes. **senior-dev uses git only if it is there.** A folder with no git history — a plain
@@ -1050,7 +1083,7 @@ after the run's record folder (such as `20260924-150405.000000`). That folder al
 `delegate-program.json`, with the instant senior-dev's process started and the instant it
 ended.
 
-## senior-dev's flags — run, --variant, --in-place, --high, --max-cost
+## senior-dev's flags — run, --variant, --in-place, --high, --verify-test, --max-cost
 
 `codeaf senior-dev <brief>` is `codeaf senior-dev run -- <brief>`. codeaf gives every
 program it carries four flags:
@@ -1073,6 +1106,9 @@ senior-dev's own flags on `run`:
   bare OpenRouter id, service-prefixed id, or short `/crew` model word;
 - `--asked` — the `--high` models were chosen by name, so one senior-dev cannot
   size ends the run before its first call rather than being skipped;
+- `--verify-build CMD`, `--verify-test CMD` — the project's own build or test command,
+  run by senior-dev on the submitted tree instead of the one it would have found (see
+  how senior-dev finds a project's build and tests);
 - `--frontier` — accepted, and changes nothing: no call senior-dev makes uses that tier;
 - `--crew` — the models came from a conversation's crew: one its catalog cannot size is
   left out instead of failing the run. codeaf passes it with the crew's models.

internal/manual/chat_test.go

@@ -3394,6 +3394,33 @@ func TestTheCallLogPageSaysWhyAFigureIsMissingFromARow(t *testing.T) {
 	t.Fatalf("the question does not reach a section that says %q; a row missing a figure reads as a broken row", said)
 }
 
+// A senior-dev run on a project whose build it could not find ends with a
+// sentence a person pastes back as a question, and a person with a harness of
+// their own asks how to hand it over. Both reach the section that names the
+// flags, and the CMake shape reaches the section that names its default.
+func TestTheUndiscoveredBuildQuestionsReachTheVerifyFlags(t *testing.T) {
+	for _, probe := range []struct {
+		asked string
+		says  string
+	}{
+		{"senior-dev says no build entrypoint could be discovered", "--verify-test"},
+		{"how do I tell senior-dev which command runs my tests", "--verify-test"},
+		{"senior-dev failed verification on my fuzz target harness", "--verify-build"},
+		{"does senior-dev build a cmake project and run ctest", "cmake-build"},
+	} {
+		found := false
+		for _, section := range Chat().Search(probe.asked, DefaultResults) {
+			if section.Page == "senior-dev" && strings.Contains(section.Body, probe.says) {
+				found = true
+				break
+			}
+		}
+		if !found {
+			t.Errorf("%q does not reach a senior-dev section that says %q", probe.asked, probe.says)
+		}
+	}
+}
+
 // A person whose reply vanished asks in their own words, and there is exactly
 // one page that can say who ended it. The pins are the two halves the fix owes
 // them: that a stop of their own is the only silent door, and that a thinking

internal/seniordev/app/args.go

@@ -64,9 +64,14 @@ type cliArgs struct {
 	// InPlace forces the snapshot recorder: senior-dev edits the workspace
 	// without writing to any repository around it. Without it, the snapshot
 	// recorder is still chosen wherever there is no git history to use.
-	InPlace  bool
-	MaxCost  *float64
-	MaxHours *float64
+	InPlace bool
+	// VerifyBuild and VerifyTest are the build and test the person named as
+	// the project's own check, each in place of discovery for its kind; empty
+	// leaves that kind to discovery (fullverification/declared.go).
+	VerifyBuild string
+	VerifyTest  string
+	MaxCost     *float64
+	MaxHours    *float64
 }
 
 // CrewModel is a crew seat's model as a pool entry: the id filed under the

internal/seniordev/app/full_verification_declared_test.go

@@ -0,0 +1,120 @@
+//go:build !windows
+
+package app
+
+import (
+	"context"
+	"io"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"github.com/Agent-Field/codeaf/internal/seniordev/session/fullverification"
+)
+
+// These tests describe the project's own check named on the command line
+// (`--verify-build`, `--verify-test`): the shape of a fuzz target or a
+// benchmark rig, whose command lives beside the checkout rather than in it.
+
+func declaredVerificationRunner(t *testing.T, args cliArgs, files map[string]string) *pipeline {
+	t.Helper()
+	workspace := t.TempDir()
+	for name, content := range files {
+		if err := writeFile(filepath.Join(workspace, name), content); err != nil {
+			t.Fatal(err)
+		}
+	}
+	runner := newPipeline(args, workspace, pipelineDeps{
+		Events: newEventWriter(io.Discard), Notes: io.Discard,
+	})
+	t.Cleanup(runner.runtime.Close)
+	return runner
+}
+
+// A DECLARED COMMAND IS RUN BY SENIOR-DEV ITSELF, IN PLACE OF DISCOVERY. The
+// command here lives outside the workspace, as a harness's would, and leaves a
+// mark to prove it ran; the Makefile's own red test is never run, because the
+// person said what this project's test is.
+func TestADeclaredTestRunsInPlaceOfTheDiscoveredOne(t *testing.T) {
+	harness := t.TempDir()
+	mark := filepath.Join(harness, "ran")
+	script := filepath.Join(harness, "validate.sh")
+	if err := os.WriteFile(script, []byte("#!/bin/sh\ntouch '"+mark+"'\n"), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	runner := declaredVerificationRunner(t, cliArgs{VerifyTest: script}, map[string]string{
+		"Makefile": "build:\n\t@true\ntest:\n\texit 1\n",
+	})
+	verification := runner.runProjectVerification(context.Background())
+	if verification.Failed != nil {
+		t.Fatalf("a green declared test failed verification: %#v (%s)", verification.Failed, verification.Failure)
+	}
+	if _, err := os.Stat(mark); err != nil {
+		t.Fatal("the declared command never ran")
+	}
+	if !soloVerificationPassed(&verification) {
+		t.Fatalf("a green build and a green declared test did not pass: %#v", verification)
+	}
+	sources := []string{}
+	for _, command := range verification.Commands {
+		evidence, _ := command.(map[string]any)
+		sources = append(sources, evidence["source"].(string))
+		if evidence["cmd"] == "make test" {
+			t.Fatal("the discovered test ran although a test was declared")
+		}
+	}
+	if strings.Join(sources, ",") != "Makefile#build,"+fullverification.DeclaredTestSource {
+		t.Fatalf("sources = %v, want the discovered build and the declared test", sources)
+	}
+	if !strings.Contains(verification.Prompt, "named by the person who started this run") {
+		t.Fatalf("the evidence the model reads does not say where the check came from:\n%s", verification.Prompt)
+	}
+}
+
+// A declared command is judged exactly as a discovered one: red is a failure,
+// and it is the command's failure, never a missing-entrypoint one.
+func TestARedDeclaredCommandFailsVerification(t *testing.T) {
+	runner := declaredVerificationRunner(t, cliArgs{VerifyBuild: "true", VerifyTest: "exit 3"},
+		map[string]string{"NOTES.txt": "a harness-shaped project\n"})
+	verification := runner.runProjectVerification(context.Background())
+	if verification.Failed == nil {
+		t.Fatal("a declared test exiting 3 passed verification")
+	}
+	if verification.Failed.Source != fullverification.DeclaredTestSource || missingEntrypointFailure(verification) {
+		t.Fatalf("Failed = %#v, want the declared test's own failure", verification.Failed)
+	}
+}
+
+// The strict preamble holds for a declared command too, so the person's
+// command cannot be passed by a masked pipeline any more than a discovered one.
+func TestADeclaredCommandRunsUnderTheStrictPreamble(t *testing.T) {
+	runner := declaredVerificationRunner(t, cliArgs{VerifyBuild: "true", VerifyTest: "false | cat"},
+		map[string]string{"NOTES.txt": "a harness-shaped project\n"})
+	if verification := runner.runProjectVerification(context.Background()); verification.Failed == nil {
+		t.Fatal("a pipeline whose first command fails passed verification")
+	}
+}
+
+// THE 5C2 SHAPE. A header-only CMake library used to fail for want of a
+// command; it now has CMake's own, and nothing in it is a missing entrypoint.
+// The commands are not run here — this machine may have no cmake — only
+// planned, which is where the old failure was decided.
+func TestAHeaderOnlyCMakeLibraryIsNoLongerMissingAnEntrypoint(t *testing.T) {
+	runner := declaredVerificationRunner(t, cliArgs{}, map[string]string{
+		"CMakeLists.txt":              "cmake_minimum_required(VERSION 3.20)\nproject(lib CXX)\ninclude(CTest)\n",
+		"src/lib.hpp":                 "#pragma once\n",
+		"extras/tests/CMakeLists.txt": "add_test(NAME smoke COMMAND true)\n",
+	})
+	plan := newProjectVerificationRun(runner, context.Background()).plan
+	if !planHasKind(plan, fullverification.KindBuild) || !planHasKind(plan, fullverification.KindTest) {
+		t.Fatalf("plan = %#v, want CMake's build and test", plan)
+	}
+	for _, entrypoint := range plan.Entrypoints {
+		// The build tree is the run's own folder, which no recorder reads, so
+		// configuring the project never puts a build directory in the answer.
+		if !strings.Contains(entrypoint.Command, seniorDevDataDirectory+"/") {
+			t.Fatalf("%q builds outside %s/", entrypoint.Command, seniorDevDataDirectory)
+		}
+	}
+}

internal/seniordev/app/full_verification_run.go

@@ -42,18 +42,32 @@ type verificationObservation struct {
 }
 
 func newProjectVerificationRun(runner *pipeline, ctx context.Context) *projectVerificationRun {
+	// What the person named with --verify-build and --verify-test replaces
+	// discovery for its kind (fullverification/declared.go).
+	plan := fullverification.Discover(runner.workspace).Declare(fullverification.Declared{
+		Build: runner.args.VerifyBuild, Test: runner.args.VerifyTest,
+	})
+	lines := []string{
+		"# Independent full project verification",
+		"senior-dev independently discovered and ran the standard project entrypoints",
+		"below in fresh Bash subprocesses. These are process-derived command/exit",
+		"observations, not the model's claims. Consult them, but still run and cite",
+		"your own fresh verification commands.",
+	}
+	for _, entrypoint := range plan.Entrypoints {
+		if entrypoint.IsDeclared() {
+			lines = append(lines, "A command whose source is "+fullverification.DeclaredBuildSource+
+				" or "+fullverification.DeclaredTestSource+" was named by the person who started this run as the",
+				"project's own check, in place of discovery; it is the one this run is judged by.")
+			break
+		}
+	}
 	return &projectVerificationRun{
 		runner: runner,
 		ctx:    ctx,
-		plan:   fullverification.Discover(runner.workspace),
+		plan:   plan,
 		result: projectVerificationResult{Commands: []any{}},
-		lines: []string{
-			"# Independent full project verification",
-			"senior-dev independently discovered and ran the standard project entrypoints",
-			"below in fresh Bash subprocesses. These are process-derived command/exit",
-			"observations, not the model's claims. Consult them, but still run and cite",
-			"your own fresh verification commands.",
-		},
+		lines:  lines,
 		issues: []string{},
 	}
 }

internal/seniordev/app/run.go

@@ -72,6 +72,12 @@ type Options struct {
 	// InPlace edits the folder without git: no commits, no refs, and the run's
 	// checkpoints kept outside it.
 	InPlace bool
+	// VerifyBuild and VerifyTest are the commands the person named as the
+	// project's own build and test (`--verify-build`, `--verify-test`). senior-dev
+	// runs each itself on the frozen tree in place of the one discovery would
+	// have chosen; empty leaves that kind to discovery.
+	VerifyBuild string
+	VerifyTest  string
 	// Crew says the pools came from the crew of the conversation that started
 	// the run (`--crew`), not from a person typing them: a model the catalog
 	// cannot size is dropped with a note, and a --high left empty routes on
@@ -130,6 +136,7 @@ func runWith(ctx context.Context, host delegate.Host, options Options, notes io.
 	args := cliArgs{
 		High: options.High, Low: options.Low, Frontier: options.Frontier,
 		Variant: options.Variant, InPlace: options.InPlace,
+		VerifyBuild: options.VerifyBuild, VerifyTest: options.VerifyTest,
 	}
 	if len(splitPool(args.High)) == 0 {
 		return refused("--high names no model, and the coder needs one to route on")

internal/seniordev/seniordev.go

@@ -186,16 +186,22 @@ func bindRun(fs *flag.FlagSet) delegate.Body {
 	frontier := fs.String("frontier", "", "models for the frontier tier (no call uses it)")
 	crew := fs.Bool("crew", false, "the models came from codeaf's crew: skip any it cannot size")
 	asked := fs.Bool("asked", false, "the --high models were asked for by name; none is skipped")
+	// The project's own check, named by whoever started the run, for a project
+	// whose build or test discovery cannot find (fullverification/declared.go).
+	verifyBuild := fs.String("verify-build", "", "the project's build command, run on the frozen tree in place of the one discovered")
+	verifyTest := fs.String("verify-test", "", "the project's test command, run on the frozen tree in place of the one discovered")
 	return func(ctx context.Context, host delegate.Host, args []string) error {
 		run(ctx, host, app.Options{
-			Goal:     strings.Join(args, " "),
-			High:     *high,
-			Low:      *low,
-			Frontier: *frontier,
-			Variant:  *variant,
-			InPlace:  *inPlace,
-			Crew:     *crew,
-			Asked:    *asked,
+			Goal:        strings.Join(args, " "),
+			High:        *high,
+			Low:         *low,
+			Frontier:    *frontier,
+			Variant:     *variant,
+			InPlace:     *inPlace,
+			VerifyBuild: *verifyBuild,
+			VerifyTest:  *verifyTest,
+			Crew:        *crew,
+			Asked:       *asked,
 		}, os.Stderr)
 		return nil
 	}