senior-dev: keep the store in the run's record folder, write back a removed .senior-dev, and --state-dir

docs/changes/unreleased/1800-senior-dev-state-dir.md

@@ -0,0 +1,30 @@
+---
+kind: fixed
+title: senior-dev keeps its session store out of the folder and writes back a removed .senior-dev
+pr: 1800
+surface: [engine, docs]
+invalidates:
+  - "A run whose `.senior-dev` was a link to a store elsewhere ended on the next model turn after the link went away, with `open <folder>/.senior-dev/projection.lock: no such file or directory`, while the store itself was whole. senior-dev now resolves the store's path once when it opens it, so the database, the records and the lock all name the store itself, and a link that goes away no longer ends the run."
+  - "The session database and conversation lived in the folder's `.senior-dev` and were moved into the task's record when the run ended. A run codeaf carries now keeps them in its record folder from the start, `<record>/store` (`store.1`, `store.2` for a later run of the same task), and the folder's `.senior-dev` holds only the brief, the checklist, the pinned command, the messages taken and the output set aside. They are in `.senior-dev` only when codeaf names no record folder."
+  - "`codeaf senior-dev run --state-dir DIR`, or `SENIOR_DEV_STATE_DIR`, keeps the store in DIR instead of the record folder, where it stays; a DIR inside the folder, other than under its own `.senior-dev/`, is refused before anything is made or spent."
+  - "A `.senior-dev` removed mid-run took the brief, the checklist and the pinned command with it. senior-dev now writes each one back after the step that removed it, as it last read it, says so on the run's page (`wrote .senior-dev/checklist.md back after something removed it`) and carries on."
+---
+
+Two CyberGym runs lost `.senior-dev` to the model's own last command. On openssl
+the folder's `.senior-dev` was a link to a store outside it, and OpenSSL 1.1.0's
+`make clean` deletes every link in the tree; the lock was reopened through the
+link for every write, so one failed `open()` ended the run unsubmitted after
+$0.64 while the store was collected intact. On arduinojson `.senior-dev` was a
+real directory, and the rig's `validate.py` began `sudo rm -rf /src`, taking the
+store and the conversation with it. Neither deleting step reached the action
+log, because the store died before the step could be written.
+
+A store that is itself removed or emptied still ends the run, now saying `its
+store <dir> was removed while the run was working, with the conversation in it`.
+The store holds its directory, its database and its records open and compares
+them with what is at their paths before every write and every read of the
+conversation: in a fresh directory the next turn would make the session again
+under the same id and the model would carry on from a conversation that starts
+where the removal happened. Holding them open also keeps ext4 and overlayfs from
+handing a removed directory's inode number to the next one made at the same
+path.

internal/manual/chat/senior-dev.md

@@ -613,17 +613,52 @@ and one rescue to **250 MiB** total. If a file or the total exceeds the limit,
 senior-dev refuses the restore before changing the folder; the ending says it
 could not be put back and that the folder holds later changes nothing checked.
 
-## Its notes — .senior-dev, its checklist, its session database, moved out when it ends
-
-senior-dev keeps its own records in `.senior-dev/` in the folder it works in (its copy, in
-a repository): the brief,
-its checklist, the command it pinned, its session database and its whole conversation
-with its model. **They are moved out of your folder when the run ends or you stop it**,
-into the task's record folder beside `delegate-conversation.jsonl` (a shell run's record
-folder at a shell), and the page adds `its notes (.senior-dev/) are kept in <path>`. So
-they never end up on a branch, and the next run in that folder never reads the last
-one's checklist as its own. A `.senior-dev/` already in the folder when the run began is
-left where it is, and never ends up on a branch either.
+## Its notes — .senior-dev, its checklist, moved out when it ends; its session database in the record folder
+
+senior-dev keeps the files it and its model work from in `.senior-dev/` in the folder it
+works in (its copy, in a repository): the brief, its checklist, the command it pinned,
+the messages it was handed and the output it set aside. **They are moved out of your
+folder when the run ends or you stop it**, into the task's record folder beside
+`delegate-conversation.jsonl` (a shell run's record folder at a shell), and the page adds
+`its notes (.senior-dev/) are kept in <path>`. So they never end up on a branch, and the
+next run in that folder never reads the last one's checklist as its own. A `.senior-dev/`
+already in the folder when the run began is left where it is, and never ends up on a
+branch either.
+
+Its session database and its whole conversation with its model are not in your folder at
+all. From the moment the run starts they are in `store/` in that same record folder
+(`store.1`, `store.2` for a later run of the same task), and they stay there. A run given
+`--state-dir DIR` (or `SENIOR_DEV_STATE_DIR`) keeps them in that directory instead. Only
+when codeaf names no record folder do they go in `.senior-dev/`.
+
+## senior-dev stopped with projection.lock: no such file or directory — .senior-dev removed or a link to it gone mid-run, its session store, --state-dir, SENIOR_DEV_STATE_DIR
+
+senior-dev's session store — `senior-dev.db`, the `storage/` folder of its records and
+`projection.lock` — is kept out of the folder it works in: in `store/` in the run's
+record folder, or where `--state-dir DIR` or `SENIOR_DEV_STATE_DIR` says (the flag
+wins). With no record folder and neither of those, it is in `.senior-dev/`. **It opens
+the store once, by its real path**, so a link on the way to it that goes away no longer
+matters. It used to live in `.senior-dev/`, reopened through it for every write,
+and a run whose `.senior-dev` link was deleted ended with
+`open <folder>/.senior-dev/projection.lock: no such file or directory` while the store
+itself was whole.
+
+**A `.senior-dev/` removed mid-run is written back.** What the work runs can empty the
+folder — a benchmark's reset script, `make clean`, `rm -rf` of the tree. After each step
+its model takes, senior-dev puts back the brief, the checklist, the pinned command and the
+messages it was handed, as it last read them, and carries on; the page says
+`wrote .senior-dev/checklist.md back after something removed it`. The session store is
+elsewhere and is not affected.
+
+**A store that is itself removed or emptied ends the run**, because its conversation went
+with it; senior-dev never carries on in an empty one made in its place. It stops, saying
+`its store <dir> was removed while the run was working, with the conversation in it`.
+
+What `--state-dir` or `SENIOR_DEV_STATE_DIR` names is made if it is missing. **Use a new
+one for each run**: runs in folders without git share one project, so a shared one mixes
+their sessions. One inside the folder is refused before anything
+is made or spent, because its checkpoints and the change it hands in would carry the
+store; the folder's own `.senior-dev/` is the exception.
 
 ## Where does senior-dev put its work — its own branch, in a copy of its own, not merged, one commit
 
@@ -1114,7 +1149,7 @@ after the run's record folder (such as `20260924-150405.000000`). That folder al
 `delegate-program.json`, with the instant senior-dev's process started and the instant it
 ended.
 
-## senior-dev's flags — run, --variant, --in-place, --high, --verify-test, --max-cost
+## senior-dev's flags — run, --variant, --in-place, --high, --verify-test, --state-dir, --max-cost
 
 `codeaf senior-dev <brief>` is `codeaf senior-dev run -- <brief>`. codeaf gives every
 program it carries four flags:
@@ -1140,6 +1175,7 @@ senior-dev's own flags on `run`:
 - `--verify-build CMD`, `--verify-test CMD` — the project's own build or test command,
   run by senior-dev on the submitted tree instead of the one it would have found (see
   how senior-dev finds a project's build and tests);
+- `--state-dir DIR` — its session store in DIR, not the run's record folder (`SENIOR_DEV_STATE_DIR`);
 - `--frontier` — accepted, and changes nothing: no call senior-dev makes uses that tier;
 - `--crew` — the models came from a conversation's crew: one its catalog cannot size is
   left out instead of failing the run. codeaf passes it with the crew's models.

internal/manual/chat_test.go

@@ -1168,6 +1168,8 @@ func TestTheChatManualAnswersTheQuestionsPeopleAsk(t *testing.T) {
 		{"why can't I run senior-dev in a folder inside the one another run is working in", "senior-dev"},
 		{"can I run senior-dev in a folder that is not a git repo", "senior-dev"},
 		{"where do senior-dev's notes go", "senior-dev"},
+		{"senior-dev crashed with projection.lock no such file or directory", "senior-dev"},
+		{"keep senior-dev's session database outside the folder", "senior-dev"},
 		{"why can't codeaf edit files while senior-dev is working", "senior-dev"},
 		{"my task was refused because senior-dev is working in the folder", "senior-dev"},
 		{"the delegate was refused because of uncommitted changes", "delegates"},
@@ -3424,6 +3426,39 @@ func TestTheUndiscoveredBuildQuestionsReachTheVerifyFlags(t *testing.T) {
 	}
 }
 
+// A senior-dev run whose .senior-dev went away under it ended with a sentence a
+// person pastes back as a question, and whoever starts it from a benchmark rig
+// asks how to keep its store out of the folder. Both reach the section that
+// names the flag, and the ending a removed store now gives reaches the section
+// that explains it. Where the database lives now, and what happens when the
+// work empties .senior-dev, are asked in a person's words too.
+func TestTheVanishedStoreQuestionsReachTheStateDir(t *testing.T) {
+	for _, probe := range []struct {
+		asked string
+		says  string
+	}{
+		{"senior-dev crashed with projection.lock no such file or directory", "--state-dir"},
+		{"open .senior-dev/projection.lock: no such file or directory", "--state-dir"},
+		{"keep senior-dev's session database outside the folder", "SENIOR_DEV_STATE_DIR"},
+		{"what does senior-dev's --state-dir do", "--state-dir"},
+		{"senior-dev says its store was removed while the run was working", "was removed while the run was working"},
+		{"where does senior-dev keep its session database", "store/"},
+		{"senior-dev's .senior-dev folder was deleted in the middle of a run", "written back"},
+		{"make clean deleted .senior-dev while senior-dev was working", "puts back the brief"},
+	} {
+		found := false
+		for _, section := range Chat().Search(probe.asked, DefaultResults) {
+			if section.Page == "senior-dev" && strings.Contains(section.Body, probe.says) {
+				found = true
+				break
+			}
+		}
+		if !found {
+			t.Errorf("%q does not reach a senior-dev section that says %q", probe.asked, probe.says)
+		}
+	}
+}
+
 // A person whose reply vanished asks in their own words, and there is exactly
 // one page that can say who ended it. The pins are the two halves the fix owes
 // them: that a stop of their own is the only silent door, and that a thinking

internal/seniordev/actions.go

@@ -268,6 +268,14 @@ func presentStage(stage, status string, facts stageFacts) (delegate.Shown, bool)
 		return delegate.Shown{Text: "stopped without handing in its work"}, true
 	case "implement/steered":
 		return steered(facts), true
+	case "implement/notes-rewritten":
+		// Something the work ran removed the file (app/notes_kept.go); the
+		// run wrote it back as it last read it and carried on.
+		text := "wrote its notes back after something removed them"
+		if file := facts.text("file"); file != "" {
+			text = "wrote " + file + " back after something removed it"
+		}
+		return delegate.Shown{Text: text}, true
 
 	case "compaction-capacity/pinned":
 		text := "learned how much its model can hold"

internal/seniordev/actions_test.go

@@ -221,3 +221,18 @@ func TestAGuessedWindowReadsOnTheRunsPage(t *testing.T) {
 		t.Fatalf("shown = %+v, %v", shown, ok)
 	}
 }
+
+// A NOTE FILE WRITTEN BACK SAYS SO on the run's page, naming the file when the
+// record does: something the work ran removed it, and the run put it back as
+// it last read it (app/notes_kept.go).
+func TestANoteWrittenBackReadsOnTheRunsPage(t *testing.T) {
+	read := Program.Reader()
+	named, ok := read(stage("implement", "notes-rewritten", map[string]any{"file": ".senior-dev/checklist.md", "bytes": 43}))
+	if !ok || named.Text != "wrote .senior-dev/checklist.md back after something removed it" {
+		t.Fatalf("shown = %+v, %v", named, ok)
+	}
+	unnamed, ok := read(stage("implement", "notes-rewritten", nil))
+	if !ok || unnamed.Text != "wrote its notes back after something removed them" {
+		t.Fatalf("shown = %+v, %v", unnamed, ok)
+	}
+}

internal/seniordev/app/args.go

@@ -70,6 +70,13 @@ type cliArgs struct {
 	// leaves that kind to discovery (fullverification/declared.go).
 	VerifyBuild string
 	VerifyTest  string
+	// StateDir is where the session store lives when the flag or the variable
+	// named it, resolved through its links ([stateDirectory]). Empty leaves it
+	// to RecordStore, where a directory of the launch's own is made when the
+	// store opens ([claimRecordStore]); with both empty the store stays in the
+	// workspace's own .senior-dev.
+	StateDir    string
+	RecordStore string
 	MaxCost     *float64
 	MaxHours    *float64
 }

internal/seniordev/app/config_live_test.go

@@ -73,7 +73,7 @@ func TestProjectConfigChangesLiveRuntimePermissionsAndInstructions(t *testing.T)
 	if err != nil {
 		t.Fatal(err)
 	}
-	runtime := newConfiguredRuntime(workspace, &capturingBackend{}, cfg)
+	runtime := newConfiguredRuntime(workspace, "", &capturingBackend{}, cfg)
 	defer runtime.Close()
 
 	// Project instructions reach the live system prompt service.
@@ -246,7 +246,7 @@ func TestSeniorDevConfigDirFeedsRegistryFormatterContract(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	runtime := newConfiguredRuntime(workspace, &capturingBackend{}, cfg)
+	runtime := newConfiguredRuntime(workspace, "", &capturingBackend{}, cfg)
 	defer runtime.Close()
 	target := filepath.Join(workspace, "sample.fmtx")
 	input, _ := json.Marshal(map[string]any{"filePath": target, "content": "unformatted\n"})