Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 17 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,23 @@ version is 0, a minor bump may break the API.
is checked and decoded into `T`, and a mismatch is repaired like a failure.
- Approval: `Capability.NeedsApproval`, the `(codemode.needs_approval)` method
option and `gosource.NeedsApproval()`. A run holds that write and every later
one and returns `*agent.Pending`; `agent.PlanStore`, `agent.NewMemoryPlans` and
`agent.ApplyPlan(ctx, mode, store, owner, id)` apply a plan once, by ID, for its
owner. A write needing approval fails rather than runs outside `agent.Run`.
one and returns `*agent.Pending`, applied through `agent.Plans` below. A write
needing approval fails rather than runs outside `agent.Run`.
- Durable plans: `agent.NewPlans(store, PlanOptions{TTL, OnEvent})` is the
approval API — `List` (the inbox), `Get`, `Approve`, `Reject` and `Resume`. A
plan has a status (`pending`, `applying`, `applied`, `failed`, `rejected`,
`expired`), an expiry, the request and run that made it, each applied call's
result, and a history of every change. Decisions are a compare-and-set, so a
plan applies once across replicas; expiry is checked on read; a crash while
applying resumes from the first call without a result. Errors name the plan
and its state, wrapping `ErrNoPlan`, `ErrPlanExpired` or `ErrPlanDecided`.
- `agent.PlanStore` (`Create`, `Get`, `List`, `Update`), `agent.NewMemoryPlanStore`,
`agent/sqlplans` for SQLite and Postgres on `database/sql`, and
`agent/plantest.Run`, the contract every plan store passes.
- `Options.OnEvent` and `agent.Event`: one event per step of a run or plan, each
with the run ID. `agent.LogEvents(logger)` sends them to `log/slog`.
- `Result.RunID`, and `*agent.Error{RunID, Stage}` wrapping every error
`agent.Run` returns except `*Pending`.
- `capability/jsonschema`: the JSON Schema converter `mcpsource` used, exported.
- `codemode/sourcetest.Run`: the contract every capability source passes.
- `examples/todo` serves its capabilities from Go functions by default;
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Flow: a source (typed Go functions via `capability/gosource`, protos via `buf ge
| `capability.Store` | `capability/sqlitestore`, `capability/pgstore` |
| `program.Cache` | `program.New` (semantic), `program.NewExactCache` |
| `program.Store` | `program.NewMemoryStore`, `capability/sqlitestore`, `capability/pgstore` |
| `agent.PlanStore` | `agent.NewMemoryPlans` |
| `agent.PlanStore` | `agent.NewMemoryPlanStore`, `agent/sqlplans` (SQLite and Postgres on `database/sql`) |
| `llm.Client`, `llm.Streamer` | `llm/claude`, `llm/openai` |
| `sandbox.Runner` | `sandbox/goja`, `sandbox/starlark`, `sandbox/lua`, `sandbox/subprocess` |
| `sandbox.Isolation` (registered) | `sandbox/subprocess` (`process`); in-process is built in |
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ test-postgres:
-e POSTGRES_DB=codemode -p 55432:5432 pgvector/pgvector:pg17
sleep 4
CAPABILITY_TEST_POSTGRES_DSN=postgres://postgres:test@localhost:55432/codemode?sslmode=disable \
go test ./capability/pgstore -run TestPostgresSearch -count=1; \
go test ./capability/pgstore ./agent/sqlplans -run 'TestPostgres' -count=1; \
status=$$?; docker stop codemode-pg >/dev/null; exit $$status

# Documentation site. Self-contained under docs/: its own package.json and
Expand Down
11 changes: 6 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,14 +46,15 @@ mode, err := codemode.FromSource(source, capability.Caller{Roles: user.Roles})
model, err := llm.NewFromEnv() // ANTHROPIC_API_KEY or OPENAI_API_KEY
runner, err := sandbox.NewFromEnv(sandbox.Options{}) // in-process JS by default
revenue, result, err := agent.RunAs[Revenue](ctx, mode, model, "how much have paid orders brought in?",
agent.Options{Runner: runner, Plans: plans, PlanOwner: user.ID})
agent.Options{Runner: runner, Plans: plans, PlanOwner: user.ID, OnEvent: agent.LogEvents(logger)})

// 4. a refund waits for a person
// 4. a refund waits for a person, in your database, until they decide
var pending *agent.Pending
if errors.As(err, &pending) {
// show pending.Calls; on approval:
calls, err := agent.ApplyPlan(ctx, mode, plans, user.ID, pending.ID)
// send pending.ID and pending.ExpiresAt to the client; later, on approval:
plan, err := plans.Approve(ctx, mode, user.ID, pending.ID)
}
// every other error is an *agent.Error carrying result.RunID and the stage that failed
```

No codegen and no `switch` over RPC names. Every call the program makes is
Expand Down Expand Up @@ -119,7 +120,7 @@ Full docs are a VitePress site under [`docs/`](docs/content).
- [Development and production](docs/content/environments.md) · [Running the worker](docs/content/worker.md) — same binary, different isolation
- [What your binary links](docs/content/dependencies.md) — the blank imports, and why
- [Permissions](docs/content/permissions.md) · [Roles and tenants](docs/content/tenants.md) — roles in the contract, rows in your code
- [Types and validation](docs/content/types.md) · [Approving writes](docs/content/approval.md)
- [Types and validation](docs/content/types.md) · [Approving writes](docs/content/approval.md) · [Observability](docs/content/observability.md)
- [Writes and retries](docs/content/writes.md) · [Caching programs](docs/content/caching.md)

**Grow it**
Expand Down
96 changes: 63 additions & 33 deletions agent/agent.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,24 @@ type Options struct {
OnCall func(codemode.Call)
// Stops repair attempts once spent; zero disables. Only providers that report usage count.
MaxTokens int64
// Plans keeps writes held for approval so they can be applied by ID; nil returns them in Pending only.
Plans PlanStore
// PlanOwner scopes a saved plan to one caller, such as a user ID; ApplyPlan must pass the same.
// Plans keeps writes held for approval until someone decides; nil returns them in Pending only.
Plans *Plans
// PlanOwner is who may decide a held plan, such as a user ID.
PlanOwner string
// OnEvent receives every step of the run, each carrying Result.RunID. It must not block.
OnEvent func(Event)

// Set by RunAs: checks the returned value, so a mismatch is repaired like a failure.
check func(any) error
runID string
}

func (options Options) emit(event Event) {
if options.OnEvent != nil {
event.Time = time.Now()
event.RunID = options.runID
options.OnEvent(event)
}
}

const (
Expand All @@ -66,6 +77,8 @@ func (options Options) checked() (Options, error) {
}

type Result struct {
// RunID is on every event and error of this run, and on the plan it held.
RunID string
Answer any
Source string
// Empty on a cache hit.
Expand Down Expand Up @@ -112,36 +125,53 @@ func Run(ctx context.Context, mode codemode.CodeMode, model llm.Client, request
if err != nil {
return Result{}, err
}
options.runID = newID(8)
options.emit(Event{Kind: EventRunStarted})

runner := options.Runner
var timing Timing
meter := &llm.Meter{}
ctx = llm.WithMeter(ctx, meter)
stage := StageCache
finish := func(result Result, err error) (Result, error) {
timing.Total = time.Since(started)
result.RunID = options.runID
result.Timing = timing
result.Usage = meter.Usage()
return result, err
var pending *Pending
switch {
case errors.As(err, &pending):
options.emit(Event{Kind: EventRunFinished, PlanID: pending.ID, Count: len(pending.Calls), Duration: timing.Total})
return result, err
case err != nil:
options.emit(Event{Kind: EventRunFailed, Stage: stage, Duration: timing.Total, Err: err.Error()})
return result, &Error{RunID: options.runID, Stage: stage, Err: err}
}
options.emit(Event{Kind: EventRunFinished, Duration: timing.Total})
return result, nil
}

key, caching := cacheKey(mode, runner, request, options)
if caching {
if result, hit, err := fromCache(ctx, mode, key, options, &timing); hit || err != nil {
stage = StageRun
return finish(result, err)
}
}

stage = StageSearch
search := time.Now()
matches, err := mode.Search(ctx, request, options.Limit)
timing.Search = time.Since(search)
if err != nil {
return finish(Result{}, fmt.Errorf("search: %w", err))
return finish(Result{}, err)
}
options.emit(Event{Kind: EventSearchDone, Count: len(matches), Duration: timing.Search})
if len(matches) == 0 {
return finish(Result{}, errors.New("search matched no capabilities"))
return finish(Result{}, errors.New("search matched no capabilities; check the capability descriptions, or the caller's roles"))
}

result, err := generate(ctx, mode, model, request, matches, options, &timing)
result, err := generate(ctx, mode, model, request, matches, options, &timing, &stage)
if err != nil {
return finish(result, err)
}
Expand All @@ -153,7 +183,7 @@ func Run(ctx context.Context, mode codemode.CodeMode, model llm.Client, request
}

// Sandbox errors are the model's to fix; transport errors are not.
func generate(ctx context.Context, mode codemode.CodeMode, model llm.Client, request string, matches []capability.Match, options Options, timing *Timing) (Result, error) {
func generate(ctx context.Context, mode codemode.CodeMode, model llm.Client, request string, matches []capability.Match, options Options, timing *Timing, stage *string) (Result, error) {
system := options.Runner.Prompt(matches)
if instructions := strings.TrimSpace(options.Instructions); instructions != "" {
system += "\nAdditional instructions:\n" + instructions + "\n"
Expand All @@ -165,24 +195,29 @@ func generate(ctx context.Context, mode codemode.CodeMode, model llm.Client, req
for attempt := 1; attempt <= options.Attempts; attempt++ {
result.Attempts = attempt

*stage = StageGenerate
generating := time.Now()
response, err := complete(ctx, model, system, prompt, options.OnChunk)
timing.Generate += time.Since(generating)
took := time.Since(generating)
timing.Generate += took
if err != nil {
return result, err
}
result.Source = Program(response)
options.emit(Event{Kind: EventProgramWritten, Attempt: attempt, Duration: took})

*stage = StageRun
answer, recorder, err := run(ctx, mode, result.Source, options, timing)
result.Calls = append(result.Calls, recorder.Calls()...)
if err == nil {
result.Answer = answer
result.Planned = planned(recorder)
return result, pending(ctx, recorder, result.Planned, options)
return result, pending(ctx, recorder, result.Planned, request, options)
}
if stop := unretryable(recorder, err); stop != nil {
return result, stop
}
options.emit(Event{Kind: EventProgramFailed, Attempt: attempt, Err: err.Error()})
if used := llm.UsageFrom(ctx).Total(); options.MaxTokens > 0 && used >= options.MaxTokens {
return result, fmt.Errorf("%w: %d of %d tokens used: %w", ErrTokenBudget, used, options.MaxTokens, err)
}
Expand Down Expand Up @@ -249,7 +284,16 @@ func fromCache(ctx context.Context, mode codemode.CodeMode, key program.Key, opt
}

func run(ctx context.Context, mode codemode.CodeMode, source string, options Options, timing *Timing) (any, *codemode.Recorder, error) {
recorder := &codemode.Recorder{Plan: options.Plan, OnCall: options.OnCall}
recorder := &codemode.Recorder{Plan: options.Plan, OnCall: func(call codemode.Call) {
kind := EventCallDone
if call.Planned {
kind = EventCallHeld
}
options.emit(Event{Kind: kind, Name: call.Name, Duration: call.Duration, Err: call.Error})
if options.OnCall != nil {
options.OnCall(call)
}
}}
running := time.Now()
answer, err := options.Runner.Run(codemode.WithRecorder(ctx, recorder), mode, source)
timing.Run += time.Since(running)
Expand All @@ -259,19 +303,18 @@ func run(ctx context.Context, mode codemode.CodeMode, source string, options Opt
return answer, recorder, err
}

func pending(ctx context.Context, recorder *codemode.Recorder, held []codemode.Call, options Options) error {
func pending(ctx context.Context, recorder *codemode.Recorder, held []codemode.Call, request string, options Options) error {
if !recorder.Pending() {
return nil
}
plan := &Pending{Calls: held}
if options.Plans != nil {
id, err := options.Plans.Save(ctx, options.PlanOwner, held)
if err != nil {
return fmt.Errorf("save plan: %w", err)
}
plan.ID = id
if options.Plans == nil {
return &Pending{Plan: Plan{RunID: options.runID, Request: request, Calls: held, Status: PlanPending}}
}
plan, err := options.Plans.hold(ctx, options.PlanOwner, options.runID, request, held)
if err != nil {
return err
}
return plan
return &Pending{Plan: plan}
}

func unretryable(recorder *codemode.Recorder, err error) error {
Expand All @@ -294,19 +337,6 @@ func planned(recorder *codemode.Recorder) []codemode.Call {
return writes
}

// Apply runs approved calls, including writes that need approval. It stops at the
// first failure; each call still passes validation, the registry and the transport.
func Apply(ctx context.Context, mode codemode.CodeMode, planned []codemode.Call) ([]codemode.Call, error) {
recorder := &codemode.Recorder{Approved: true}
ctx = codemode.WithRecorder(ctx, recorder)
for _, call := range planned {
if _, err := mode.Execute(ctx, call.Name, call.Input); err != nil {
return recorder.Calls(), err
}
}
return recorder.Calls(), nil
}

func complete(ctx context.Context, model llm.Client, system, prompt string, chunk func(string)) (string, error) {
if chunk != nil {
if streamer, ok := model.(llm.Streamer); ok {
Expand Down
96 changes: 96 additions & 0 deletions agent/event.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
package agent

import (
"context"
"fmt"
"log/slog"
"time"
)

// Event is one step of a run or a plan. Every event of one run carries its
// RunID; plan events carry the PlanID too, so a log line, a plan and an error
// can be joined on one key.
type Event struct {
Time time.Time
Kind string
RunID string
PlanID string
// Stage is where a run failed: cache, search, generate, or run.
Stage string
// Name is the capability, on call events.
Name string
Attempt int
Count int
Duration time.Duration
Err string
}

const (
EventRunStarted = "run.started"
EventSearchDone = "search.done"
EventProgramWritten = "program.written"
// A program that failed and goes back to the model for repair.
EventProgramFailed = "program.failed"
EventCallDone = "call.done"
EventCallHeld = "call.held"
EventRunFinished = "run.finished"
EventRunFailed = "run.failed"

EventPlanHeld = "plan.held"
EventPlanApplying = "plan.applying"
EventPlanApplied = "plan.applied"
EventPlanFailed = "plan.failed"
EventPlanRejected = "plan.rejected"
EventPlanExpired = "plan.expired"
)

// LogEvents writes each event to logger: failures at error level, the rest at info.
func LogEvents(logger *slog.Logger) func(Event) {
return func(event Event) {
attributes := []slog.Attr{slog.String("run", event.RunID)}
add := func(key, value string) {
if value != "" {
attributes = append(attributes, slog.String(key, value))
}
}
add("plan", event.PlanID)
add("stage", event.Stage)
add("capability", event.Name)
if event.Attempt > 0 {
attributes = append(attributes, slog.Int("attempt", event.Attempt))
}
if event.Count > 0 {
attributes = append(attributes, slog.Int("count", event.Count))
}
if event.Duration > 0 {
attributes = append(attributes, slog.Duration("took", event.Duration))
}
level := slog.LevelInfo
if event.Err != "" {
level = slog.LevelError
attributes = append(attributes, slog.String("error", event.Err))
}
logger.LogAttrs(context.Background(), level, event.Kind, attributes...)
}
}

// Error is every failure agent.Run returns, except *Pending: which run, and
// at which stage. errors.Is and errors.As see through it to the cause.
type Error struct {
RunID string
Stage string
Err error
}

func (failure *Error) Error() string {
return fmt.Sprintf("run %s: %s: %v", failure.RunID, failure.Stage, failure.Err)
}

func (failure *Error) Unwrap() error { return failure.Err }

const (
StageCache = "cache"
StageSearch = "search"
StageGenerate = "generate"
StageRun = "run"
)
Loading
Loading