diff --git a/docs/intent-outcome-audit.md b/docs/intent-outcome-audit.md new file mode 100644 index 0000000..03222d6 --- /dev/null +++ b/docs/intent-outcome-audit.md @@ -0,0 +1,287 @@ +# From Tasks to Persistent Intent + +## Scope and Verdict + +Audit date: 2026-09-16. Source baseline: `94fd9b3` on `origin/main`. +Evidence: source, relevant existing tests, retained local runtime records, and +GitHub issue #355 / PR #356. Historical logs are samples, not a live fleet audit. +The proposed capabilities and acceptance scenarios below are not implemented by +this document. No workers, schedules, credentials, or live objectives were changed. + +The operator's target is a persistent, autonomous system that translates human +intent into useful outcomes, including work outside software development. GitHub +can remain its workspace and project-management interface. + +Agent-OS has useful execution infrastructure, but its central contract still +follows a repository task through an agent result and GitHub updates. It does not +consistently preserve, own, and verify the human outcome across that whole path. +Adding more agents or task types alone will leave this gap intact. + +Success should mean: the requested outcome has supporting evidence at the right +destination, within delegated authority and budget. If it cannot be achieved yet, +the system retains ownership, reports the specific dependency, and knows when to +resume. Persistence includes waiting without spending model calls. + +## Findings + +### 1. The original intent can change during intake + +[`task_formatter.py`](../orchestrator/task_formatter.py) instructs the model to +infer two to four success criteria and always prefer minimal diffs. Its output +does not distinguish explicit requirements from assumptions. Context injection +adds useful objective and sprint material, but does not establish that the +interpretation matches what the person meant. + +A retained footer task illustrates this: a complaint about different links +apparently leading to the same content became a criterion about links sharing +the same domain. These require different fixes. The original intent can survive +as prose while the executable success criterion has already drifted. + +**Required:** retain the original request and revisions, identify assumptions, +record target and acceptance criteria with their sources, and ask a focused +question when ambiguity would materially change the result. Low-risk work can +proceed with disclosed assumptions; every request need not become an interview. + +### 2. No durable owner consistently carries a goal through its children + +In [`github_dispatcher.py`](../orchestrator/github_dispatcher.py), +`_try_decompose()` closes the parent epic after creating children, promotes the +first child to Ready, and sends the rest to Backlog. Splitting work therefore +closes the original issue before its aggregate outcome is verified. +`create_followup_task()` in [`queue.py`](../orchestrator/queue.py) creates a new +task with the previous next step as its goal. Ancestry and prior context exist, +but these are not an enforced parent-outcome completion rule. + +**Required:** a stable goal identity across plans, child tasks, retries, waits, +and restarts. Child completion must not close the goal. Someone can cancel or +revise a goal without leaving old children free to act under superseded intent. +Recurring duties also need a next-check condition rather than a single terminal +task or a continuously running worker. + +### 3. The workspace is too tightly coupled to the execution target + +`build_mailbox_task()` selects the configured local repository, and +`write_prompt()` tells every worker it is a coding worker confined to the current +repository. The formatter does recognize research, content, design, and browser +automation, but their execution still passes through this repository-centric +contract. `_maybe_transfer_public_website_issue()` adds a useful companion-site +heuristic; it is not general target discovery. + +[`tool_registry.py`](../orchestrator/tool_registry.py) validates configured tool +records and formats allowed tools into the prompt. The +[`runner`](../bin/agent_runner.sh) launches CLIs with approval bypasses. A tool +description in a prompt does not enforce per-action authorization. In particular, +`write_prompt()` treats available credentials as sufficient reason to publish +external posts. Credentials prove access, not permission for a particular action. + +**Required:** separate the workspace from the object being changed: deployed +site, document, account, recording, service, or several repositories. Discover +and test relevant capabilities before declaring work human-only. Enforce allowed +actions, data boundaries, and spending limits at execution, including any shell +escape path. Reuse existing approvals; bind approval to the actual action and +current goal revision. Missing authority or physical access can legitimately +require a human handoff. + +The existing trusted-author filter is useful for issue intake. Broader tool use +also needs to treat retrieved pages, messages, and files as evidence rather than +instructions that can grant authority. Test that boundary when adding tools. + +### 4. Completion has several conflicting meanings + +The queue combines `.agent_result.md`, tests, Git changes, and task-specific +checks. These are useful evidence sources, but neither a model claim nor a new +diff universally proves the requested outcome. An already delivered change may +produce no new diff; a locally created page may never reach the requested site. + +In [`github_sync.py`](../orchestrator/github_sync.py), `sync_result()` can put an +issue in Done while its PR is still open. Conversely, its terminal-issue guard +can preserve GitHub's closed state without correcting the queue's partial result. +[`pr_monitor.py`](../orchestrator/pr_monitor.py) has additional review and merge +gates, but skips dispatcher-only repositories. Mode selection changes which +stages run; the meaning of a user-visible completion needs to remain explicit. + +Issue [#355](https://github.com/kai-linux/agent-os/issues/355) is a concrete +contradiction. [PR #356](https://github.com/kai-linux/agent-os/pull/356) merged the +requested README title change on 2026-09-01. The retained worker log shows a +complete result downgraded to `partial / no_diff_produced` and a blocked queue +state. The metrics retain that partial result. The completion-notification path +requires `complete`, explaining the absent Complete message in this case. + +**Required:** a canonical lifecycle reconciled against actual outcome evidence. +Distinguish implementation prepared, reviewed, merged, deployed, and verified +when the goal requires those stages. For a document or video, use appropriate +artifact and destination checks. A script is a valid intermediate deliverable +for a video, but cannot satisfy a request for a published recording. + +### 5. Recovery and communication are not one durable transaction + +The queue has locks, stalled-process recovery, retry limits, provider cooldowns, +and structured blockers. These address real failures. The sampled April 23 +reception-viewer retry log records an immediate provider quota failure, not an +hour of productive work. Later cooldown protections exist; that historical +example does not establish that today's exact behavior is unchanged. + +Recovery still primarily operates at task/process boundaries. It cannot generally +tell whether an external action succeeded before a worker died, or resume from +a durable action receipt. Completion metrics, GitHub updates, mailbox moves, and +Telegram sends are separate operations. `send_telegram()` can fail without a +durable completion-message retry. The +[`incident router`](../orchestrator/incident_router.py) already persists pending +incident notifications, so that useful pattern should be extended rather than +ignored. + +**Required:** durable checkpoints and action receipts, explicit wait/wake +conditions, and budgets across an entire goal lineage. A retry needs a reason +it could now succeed. Persist lifecycle changes and pending notifications +atomically; reconcile GitHub and Telegram from those changes. Distinguish a live +process from recent meaningful progress. Never imply exactly-once external +effects when a provider offers no idempotency or reconciliation mechanism. + +### 6. Memory is mostly about repository work + +[`codebase_memory.py`](../orchestrator/codebase_memory.py) injects repository +summaries and recent changes, then appends information from complete results. +Objective files, sprint history, approval records, and prior attempts retain +additional context. These are useful foundations, but do not form a durable, +queryable record of a person's intent, preferences, decisions, and current facts. +For Telegram, ordinary free text is handled in the pending QA-reply path rather +than as a general goal-steering conversation. + +**Required:** distinguish user decisions, observed facts, model hypotheses, +action receipts, and reusable procedures. Retain source, time, scope, and +supersession information; check freshness when it matters. Retrieve relevant +context per action. A generated completion summary must not silently become a +verified fact. Human answers must resume the waiting goal instead of requiring +a fresh issue that loses the previous work. Apply retention and access rules so +personal context is not indiscriminately copied into public issues or prompts. + +### 7. Self-assessment rewards component presence and activity + +[`system_architect.py`](../orchestrator/system_architect.py) discovers roles from +Python filenames, jobs from function-name text, agents from configuration, and +sensors partly from metric filenames. `evaluate_system_architect()` compares +those names with the target model. Even a no-op module can satisfy presence; +the existing tests use no-op jobs to exercise this inventory. It cannot establish +that the end-to-end capability actually works. + +The tracked [strategy](../STRATEGY.md) also pursues an autonomous software +organization and GitHub adoption, allocating at least 40% of sprint capacity to +adoption/credibility work. That is a different mandate from the operator's new +goal. Adding modules under the old mandate can generate more activity without +making the system a dependable translator of intent. + +**Required:** behavioral evaluations drawn from real failed goals, with fresh +evidence, user corrections, intervention count, elapsed time, and total cost. +Self-improvement must address a reproduced failure and demonstrate improvement +on both that case and held-out cases. Missing measurement is its own gap, not +evidence that another feature is needed. Replace the old planning objective +before enabling autonomous planning for this new direction. + +## What the Retained Metrics Establish + +Read from the local `runtime/metrics/` files on the audit date: + +| Dataset | Coverage in retained file | Observation | +| --- | --- | --- | +| `agent_stats.jsonl` | 281 records, March 18 to September 1, 2026 | 206 complete, 46 partial, 29 blocked | +| `outcome_attribution.jsonl` | 70 snapshots, March 31 to April 20, 2026 | 65 inconclusive, 4 improved, 1 unchanged | + +These are different populations and time windows. Outcome snapshots are not +unique human goals, and recorded improvement is not independently established +causation. `record_metrics()` intentionally excludes some infrastructure/quota +failures for model-quality scoring. Consequently, these files do not supply an +end-to-end user-goal success rate. A missing recent snapshot also does not prove +that every live service is inactive. + +Report execution health separately from goal fulfillment and downstream impact. +A delivered research brief can be verified now while its business impact remains +unknown. Neither a merged PR nor an inconclusive adoption metric should decide +all three questions. + +## Minimum Operating Model + +Use a durable goal record as the owner of work. Keep its original request, +revision, interpreted outcome, assumptions, targets, constraints, delegated +authority, budget, and evidence requirements together. Attach plans, child +tasks, action receipts, evidence, questions, and wake conditions to that identity. +Simple tasks should take a short path through the same model. + +The control loop is: understand, inspect current reality, plan, act, observe, +verify, then complete, revise, or wait. Model workers propose plans and actions; +deterministic lifecycle rules enforce authority, limits, and evidence gates. +Before repeating an uncertain external action, reconcile what actually happened. + +Choose evidence checks before execution. Separate directly observed results from +model judgments. Subjective work needs a rubric grounded in the person's +preferences or examples, with uncertainty and review needs made explicit; a +second model's approval is not by itself proof of usefulness. + +Use one durable lifecycle store with atomic transition and notification records. +GitHub and Telegram are interfaces to it; authenticated board moves and messages +become commands, and accepted state changes project back to those interfaces. +Synchronization must not interpret its own status updates as fresh user intent. +Reconcile missed updates after outages. Do not build a second competing status +system alongside the mailbox without a migration/compatibility boundary. + +Suggested states are Backlog, Clarifying, Ready, Running, Waiting, Verifying, +Succeeded, Failed, and Cancelled. Task-level In Progress requires a current worker +lease and observable progress; a parent can remain active while its child waits, +with that reason visible. Existing board columns can represent these through +explicit status/label mappings; extra columns alone do not fix ownership. + +A human handoff preserves completed work, names the exact decision or access +needed, and states how an answer will resume execution. It is a waiting state, +not success or an invitation to run the same failed attempt indefinitely. +Missing capabilities can be acquired or delegated within the same authority and +budget; newly generated skills need tests before reuse. Arbitrary requests do +not imply unrestricted permissions or a guarantee that all work is possible. + +Retain the useful worktree/CLI execution path for coding, cost controls, +approvals, blocker taxonomy, independent PR verifier, quality harness, and +incident delivery. Adapt these behind the goal lifecycle rather than replacing +the whole system or adding another layer of agent roles. + +## Delivery Order and Acceptance Gates + +1. **Make one goal truthful end to end.** Establish stable identity, explicit + outcome evidence, lifecycle transitions, and reliable status delivery. Replay + #355 and a false-completion case. Include dispatcher-only mode. Do not expand + the autonomous workload until these cases agree across worker, queue, board, + and notifications. +2. **Prove persistence across different kinds of work.** Carry a small research + deliverable and a real recording workflow through tool discovery, multiple + steps, restart, verification, and a real wait/resume. Keep the parent open + until its outcome is satisfied. For publication without delegated access, + report a tested handoff, not a completed publication. This exercise determines + which general-purpose interfaces are actually needed. +3. **Enable bounded learning and expansion.** Align planning with the new + mandate, migrate existing task state explicitly, and gate capability claims + on behavioral evaluations. Run recovery and authority tests before increasing + concurrency or permitting autonomous tool acquisition. Evaluate cost and + user corrections alongside completion, including unsuccessful attempts. + +The following are proposed acceptance scenarios, not an existing passing suite: + +| Scenario | Required observation | +| --- | --- | +| A site request is filed in its backend workspace | Inspect ownership and the actual route; act on the right target or clarify, without inventing a local substitute. | +| Different links appear to show the same content | Verify destinations/content; do not substitute a same-domain rule. | +| An agent exits without its result file after delivering work | Inspect durable receipts and actual output; verify or report uncertainty before retrying. | +| The requested change is already merged (#355) | Verify the requested content and target revision; reconcile to success without creating a redundant diff. A merely closed issue is insufficient. | +| A worker claims complete but only produced a plan | Keep the goal unsatisfied when the required artifact or external effect is absent. | +| A goal splits into several tasks | Parent remains owned until aggregate criteria pass; unstarted children remain visible. | +| A worker or host dies after an upload | Reconcile the action receipt or remote object before retrying; do not blindly duplicate the upload. | +| Access or a human decision is needed | Persist a specific question and human-wait state, stop model retries, then resume the same goal with retained artifacts. | +| Provider quota is exhausted | Wait until a relevant condition changes, within global and per-goal budgets; no stream of misleading starts. | +| Telegram or GitHub rejects an update | Retain the pending transition, retry delivery safely, and show delivery uncertainty without changing the goal outcome. | +| A person changes or cancels the goal | Fence queued actions under the old revision, reconcile in-flight effects, and show any irreversible work already performed. | +| A goal must wait 24 hours | Survive a restart, wake when due, refresh stale facts, and continue without repeated model calls during the wait. | +| A tool is installed but unavailable or unauthorized | Report tested readiness separately from installation; refuse unauthorized effects even if credentials exist. | +| A new capability is declared supported | Pass representative outcome and injected-failure tests, not just an inventory check or self-review. | + +Verification performed for this audit: `python3 -m pytest tests/ -q` passed all +741 existing tests, including the targeted queue, GitHub sync, incident routing, +outcome attribution, and system-architect tests. Those results validate existing +test expectations; they do not establish that the proposed end-to-end acceptance +scenarios pass. Local document links and diff whitespace were checked. No live +non-coding pilot was run as part of this audit. diff --git a/docs/roadmap.md b/docs/roadmap.md index c80781c..38798d2 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,10 +1,20 @@ # Roadmap -Agent OS is not meant to stop at task execution. Its job is to bootstrap itself from a reliable execution engine into an evidence-driven, closed-loop operator that can grow products with decreasing human supervision. +Agent-OS should persistently translate human intent into verified outcomes, +including work outside software development. GitHub can remain the workspace +and project-management interface without defining the limits of execution. -That means the repository should be evolved intentionally toward Level 4 and beyond. New orchestration, planning, research, memory, and feedback systems should be judged by whether they move Agent OS up this ladder. +The [2026-09-16 intent-to-outcome audit](intent-outcome-audit.md) documents the +current gaps, reusable foundations, delivery order, and proposed acceptance +scenarios. The immediate priority is truthful completion and durable goal +ownership before expanding autonomous activity. This is a design direction, +not a claim that the runtime or its existing planning objectives have changed. -## Capability Ladder +## Earlier Capability Ladder + +This ladder describes the earlier product-growth direction. It is useful as a +capability inventory, but module presence alone does not establish readiness at +any level. Progress now needs end-to-end evidence against human goals. ### Level 1: Reliable execution engine - dispatch, queue, retries, CI, merge, memory @@ -34,4 +44,10 @@ That means the repository should be evolved intentionally toward Level 4 and bey ## Current Position -Approximately Level 2. The system already has reliable execution, persistent memory, backlog grooming, strategic planning, retrospectives, and self-healing CI remediation. The next bottleneck is evidence: richer product inspection, research, analytics, and measurable outcomes. +Execution, repository memory, backlog grooming, strategic planning, +retrospectives, and CI remediation components exist. Their presence does not +establish reliable end-to-end execution: the audit found conflicting completion +states, intent drift, premature parent closure, and limited outcome evidence. +No overall autonomy level is established. Readiness should be assessed through +the audit's behavioral acceptance gates, including recovery, authority, and +non-coding work.