feat(agent-org): add authoritative team lifecycle - #781
Closed
ShiboSheng wants to merge 11 commits into
Closed
Conversation
Establish durable Starting materialization receipts and a centralized quiescence boundary so Working teams enter Idle only from committed facts. Keep Run View pure-read, bound the global watchdog, and stop frontend polling for hidden or non-pollable teams behind one fail-closed rollout gate. Verification: - cargo clippy --all-targets -- -D warnings - Packaged Tauri Command+5 foreground, hidden, and restore lifecycle passed - Run View refresh left the isolated database byte-identical Pre-commit hook ran. Total eslint: 0, total circular: 0
Collaborator
Author
|
orgii://cloud/session/ref?v=1&org=bfa7b134-2486-45fa-81ad-a369441fafb4&owner=776dbd69-ac1d-4f72-a0d4-69cb4f2667dd&session=codexapp-rollout-2026-08-11T00-04-03-019fec6a-b4fe-7333-9faa-514f8a83850e |
With ORGII_AGENT_ORG_REDESIGN unset (production default) the send preflight hard-failed require_agent_org_redesign() for any session with a persisted org_member_id, bricking every pre-redesign org session. Gate-off now skips the Team lifecycle fence entirely: the message flows as an ordinary session turn (old behavior), and the turn claims no durable run ownership so the execute-time org promote cannot silently no-op it. Per-endpoint gate-off decisions for the remaining require sites, based on each frontend consumer: - agent_org_session_run_view -> Ok(None). READ; polled for arbitrary sessions; None is the canonical 'not an org session' reply. - agent_org_group_chat_history_page -> empty page. READ; fetched unconditionally once the Group Chat view mounts. - agent_org_run_list -> Ok(vec![]). READ; Inbox flat list renders empty. - agent_org_session_enter_intervention -> Ok(false). Called by the CLI transport for every direct user message; interventions do not exist pre-redesign, so 'nothing changed' is correct instead of a warn-logged agent_org_redesign_disabled error per send. - agent_org_session_intervention_state -> intervention: None. READ. - Kept the structured error (fail closed): pause/resume, return-to-work, send-user-message-to-member, group-chat send, plan-approval detail and respond, and the AgentOrg launch target. All are mutations (or reads reachable only through a non-null run view) that must not pretend the redesign lifecycle exists while it is disabled. Note: rollout::is_enabled() is hardwired on in cfg(test) builds, so the gate-off branches are exercised only in production configuration; the configured_enabled() unit tests continue to pin the gate parsing.
agent_delete_session required run_status == archived, but nothing in the lifecycle ever produces Archived — every org run was permanently undeletable. Deletion is now allowed for the terminal/quiescent statuses idle, failed, and archived; starting/running/paused keep the structured refusal. Idle-delete guard: an idle team must actually be quiescent when the delete commits. The plan is reloaded and revalidated inside the IMMEDIATE delete transaction (status re-check), and for idle runs the transaction additionally refuses when any in-flight (optimistic/queued/running) durable turn intent still references the run — a concurrent re-activation cannot be deleted out from under. Failed/archived teams deliberately skip the intent assertion: a crash-orphaned running intent must not make a dead team immortal. The existing in-memory runtime quiescence check (active turn / scheduler pending) continues to run before planning for all statuses. Tests: failed and idle hierarchies delete cleanly; an idle run with an in-flight intent fails closed; the running-run refusal test is unchanged.
launch_org.rs classified permanent identity failures with
.contains("identity mismatch") / .contains("Session identity"), but the
store emits 'materialization session mismatch for ...' and 'materialized
Session ... is missing for ...' — neither matched, so a permanently wrong
identity was retried forever instead of failing Starting.
Replace substring matching with stable machine prefixes defined next to
the error construction in AgentOrgRunStore:
- MATERIALIZATION_IDENTITY_MISMATCH_PREFIX now stamps all three permanent
identity failures in mark_materialization_succeeded and the certified-row
revalidation in finish_starting;
- STARTING_INPUT_CERTIFICATE_ERROR_PREFIX stamps the two permanent
initial-input certificate failures in finish_starting;
- is_materialization_identity_mismatch_error() /
is_permanent_finish_starting_error() are the only classification points.
Sweep of substring classifications introduced by this PR:
- launch_org.rs materialization receipt classification -> typed helper;
- launch/mod.rs both finish_starting sites (background launch + startup
recovery) -> is_permanent_finish_starting_error();
- launch/mod.rs initial-input payload decode classification -> shared
prefix constants + is_permanent_initial_input_payload_error(), used by
both the constructor and the classifier.
handle_background_launch_failure discarded fail_starting's Ok(false) (returned when the run already left Starting) and then marked the coordinator session failed anyway — a Running team with a failed coordinator session row, and a log that claimed the run was marked failed when it was not. When fail_starting reports no transition, the handler now reloads the actual run status and logs error-level with that status, stating explicitly that the run status was left unchanged. For a Running run the coordinator session status is also left untouched (the watchdog and the initial-dispatch recovery own retries for a live team); the launch error is still broadcast and persisted to the transcript so the failure stays visible. Non-Running outcomes keep the previous behavior.
A crash mid-turn leaves an org turn intent 'running' forever: restart
reconciliation deliberately retains running Agent Org intents, quiescence
counts them as InFlightTurnIntents, and the watchdog mapped that blocker
to {} — the team stayed Working with no repair path.
Add a bounded repair: when inspection sees a Working run whose ONLY
quiescence blockers are in-flight turn intents, it selects 'running'
intent rows older than STALE_INTENT_REPAIR_GRACE_SECS (15 min, following
the PENDING_MATERIALIZATION_GRACE_SECS pattern) into the plan. The
executor terminalizes them (running -> failed, a legal state-machine
transition) inside a writer transaction that revalidates run status,
intent status, and staleness, logging error-level per repaired intent
with the reason. Young intents are never touched, and queued canonical
initial inputs keep their own recovery owner. The next quiescence pass
idles the team.
The terminalization is performed with direct SQL in agent_core (matching
how quiescence and run deletion already address session_turn_intents from
this crate); the session-persistence state machine is respected by
construction via the status='running' guard.
Tests: wedged intent + aged clock -> plan carries the repair -> repair
marks it failed -> try_transition_working_to_idle succeeds; a young
running intent is never selected.
Transient launch errors (workspace prep, materialization, initial-input persistence, finish_starting) deferred to the next app restart while the frontend polled the Starting team forever — nothing owned retries within a session. The watchdog tick now runs a bounded Starting retry pass after the running-scan: at most STARTING_RETRY_MAX_RUNS_PER_TICK (10) starting runs older than STARTING_RETRY_GRACE_SECS (2 min; younger runs are still owned by their in-process launch task) are re-driven through the same per-run startup recovery routine, sharing the tick's 250ms deadline. A team that reaches Running this way may still hold its queued canonical initial input, so the bounded initial-dispatch pass is re-driven under the same deadline. recover_agent_org_initial_dispatches also used ? inside its loop, so one bad team (payload decode failure, send error) starved every later team on every boot. Failures are now isolated per team (log + continue), and a permanently undecodable payload on a Running run takes a terminal disposition: the queued canonical intent is rejected (a legal queued->rejected transition through the turn-intent bridge), so the input stops matching the recoverable query forever instead of being retried every boot and every tick. This was chosen over adding a new initial-input status because it needs no DDL change and the recoverable query already keys on the intent status. Tests: select_aged_starting_runs bounds, grace filtering (young runs are never selected; corrupt updated_at escalates as aged).
list_runs_by_status orders 'updated_at ASC LIMIT 100'; with more than 100 Working teams the same oldest batch was selected every tick and teams 101+ were never inspected (a stalled team's run row is not written by inspection, so its updated_at never moves it into the window). Add an in-process keyset cursor (last visited updated_at/id): each tick continues strictly after the previous batch via list_running_runs_after(), and wraps around to the front (deduplicating within the batch) when the tail is reached, so every Working team is visited across ticks regardless of population size. Test: a population larger than the batch limit is fully visited across ticks with a small limit, and successive ticks do not restart at the identical oldest batch.
A crash between reserve and commit/refund left a reservation_token behind forever; quiescence fails closed on ActiveRecoveryReservations, so the team could never idle again. Reservations are process-scoped claims by definition, so the startup one-shot now clears ALL reservation tokens unconditionally, then prunes agent_org_recovery_attempts rows whose run is no longer starting/running (missing/paused/idle/failed/archived) in LIMIT-bounded writer batches (256/batch, hard cap) so a large backlog never holds the writer lock for one long scan. The same one-shot also re-wires AgentOrgPlanApprovalStore::cancel_pending_for_terminal_or_missing_runs, which had lost its only production caller in this PR — pending approvals for dead teams are cancelled again at boot. Test: leaked tokens on running/idle/missing-run rows are all cleared; only the running team's budget row survives the prune.
Hygiene batch from the PR 781 audit:
- The copy-pasted assess-quiescence -> extract generation/work_revision ->
try_transition_working_to_idle block (4 sites: scheduler post-intent
reconcile, lifecycle turn finalization, plan-approval follow-up,
watchdog terminal candidate) is hoisted into
AgentOrgRunStore::try_reconcile_to_idle() so the certificate protocol
cannot drift per caller.
- AgentOrgRunStore::create is production-dead (launch goes exclusively
through create_starting) but is needed by unit tests and the
#![cfg(debug_assertions)] /test endpoints, which seed arbitrary-status
runs create_starting cannot express (no coordinator identity, non-
Starting statuses, NULL root). Gate it #[cfg(any(test,
debug_assertions))] — present exactly where its consumers exist,
absent from release binaries.
- Drop 'cancelled' from the last_activity_outcome CHECK: no code path
reads or writes that value ('failed' via fail_starting and 'completed'
via the idle CAS are the only writers). CREATE TABLE IF NOT EXISTS
leaves existing databases untouched; PR 799 above regenerates its DDL
manifest from the binary. The DDL string is edited minimally, no
reformatting.
resolve_session_identity computed has_persisted_agent_org_identity only when the unrelated needs_db condition forced a DB read; a fully cache-hit send (model/account/workspace/harness all in the runtime cache) skipped the probe and therefore the Agent Org lifecycle fence entirely. The probe is now unconditional: the loaded DB record answers when present; a runtime already carrying its org context answers directly; and ordinary cache-hit sessions pay one targeted indexed point read of agent_sessions.org_member_id (new session_persistence::get_org_member_id) instead of silently skipping the fence. DB errors propagate like the existing identity resolution errors.
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #756.
The existing Agent Org lifecycle does not provide an authoritative, restart-safe boundary for a long-lived Team. Initial member creation can be resumed only through loosely coupled session state, finality is inferred by multiple paths, Run View and background recovery do not have a sufficiently strict read/write boundary, and the frontend retains fallback polling for states that should be quiet.
That model cannot safely support a Team that finishes one batch of work, becomes reusable and idle, and later accepts more work. It also makes identity duplication, stale quiescence decisions, accidental read-side writes, and retained background activity difficult to reject by construction.
Solution
This PR implements the first lifecycle slice from the long-lived Team design:
Startingtogether with canonical member materialization receipts and the initial-input intent. Restart recovery continues the same(member_id, agent_id)identities instead of regenerating a Team.work_revisionstate.Working -> Idleonly through a generation-and-revision conditional transition. Stale observations become no-ops.running-only scan,LIMIT 100, and one 250 ms cooperative deadline shared by the entire tick.starting/runningallowlist. Hidden windows and the last pollable subscriber destroy the shared timer; visibility recovery performs one immediate bounded revalidation.ORGII_AGENT_ORG_REDESIGN, which remains disabled by default.The current compatibility wire value for Working remains
running.Potential risks
ORGII_AGENT_ORG_REDESIGN=1, users can reachIdlebut cannot yet submit a new Root/Group/direct Turn or mutate the Task graph from Idle. The production default must remain disabled until the later stack is complete.Running -> Idleuses a serialized generation/work_revisioncheck. A concurrent work mutation wins by changing the revision, so the idle attempt becomes a no-op; later PRs must not introduce a second lifecycle owner or bypass that revision boundary.Current implementation state
Startingowns first materialization and restart recovery. Successful construction ends inrunningwhen initial formal work exists oridlewhen it does not; unrecoverable construction ends infailedwith structured diagnostics.Idlemeans the current formal work is quiescent and the Team is retained. It is not a terminalCompletedstate.status='running') Teams. Starting recovery is a one-shot launch/startup responsibility.startingorrunningTeam.0, ortruevalues remain disabled; only1enables the internal slice.Scope boundaries and next-PR handoff
PR1 does not implement Pause/Resume redesign, Archive/Delete, the final Task FSM, additional Writers, UserDirectedWork, direct Member work, Group multi-mention, peer side quests, or the final Group read surface.
The enabled PR1-only intermediate state deliberately rejects new work from Idle:
team_idle.idlebefore writing an Inbox row.runningand returnsagent_org_run_not_mutablefor Idle.Idle -> Workingtransition occurs.PR7 is expected to add the atomic first Task graph plus
Idle -> Workingactivation. PR8 opens direct Member UserDirectedWork, PR9 opens Group targeting and peer side quests, and PR10 completes the bounded Group read surface and default rollout. Earlier follow-up PRs must extend the frozen launch roster, generation, revision, and lifecycle transition APIs established here rather than creating another state owner.Verification
Automated checks:
cargo test -p agent_core— passed on the frozen PR1 implementation before packaged acceptance.pnpm typecheck— passed before packaged acceptance.cargo clippy --all-targets -- -D warnings— passed again after fast-forwarding to the latestdevelop.agent_core,org2, andsession_persistence, ESLint 0, circular dependency count 0.git diff --check upstream/develop..HEAD— passed.Packaged Tauri acceptance used a temporary packaged
.applaunched through macOSopen, an isolatedORGII_HOME/SQLite database, isolated ports, a mock provider, and the redesign gate enabled in both frontend and backend:updated_atandwork_revisiondid not change.LIMIT 100; the shared scan used 9-25 ms of its 250 ms budget.Performance verdict: pass.
Visual screenshots are not included because this PR's meaningful acceptance evidence is lifecycle persistence, database immutability, API cadence, hidden-window behavior, and background-resource ownership rather than a visual layout change.