ProofStack

English | 한국어

ProofStack is an open-source Agent Reliability Engineering platform for observing, reproducing, evaluating, governing, and safely releasing AI agents.

Important

ProofStack is an experimental foundation, not a production release. The implemented path is real and tested, including optional PostgreSQL persistence, scoped workload API keys, the OIDC browser-session backend, bounded OTLP/HTTP trace ingestion, and retention-safe classified model and tool interaction capture. The capture path is API- and SDK-accessible and tested through encrypted artifact ownership, revocation, export, and coordinated recovery. Experimental durable replay publishes exact releases and plans, persists bounded jobs, budgets, leases, cancellation, usage, observations, and encrypted result artifacts, and launches separate worker and target processes. Framework-independent non-model evaluation primitives now provide safe applicability, digest-bound exact and JSON Schema oracles, explicit reference aggregates, and an authorization-first immutable graph repository with bounded application use cases. The graph is now persisted through PostgreSQL, exposed through an exact-version API and TypeScript SDK, and split from worker-owned evidence writes by separate database authority. Model-assisted and human evaluation now adds exact model, prompt, tool, qualification, calibration, blinding, critique, independence, counterevidence, and accountable-review records. Its service-backed reference deliberately remains ineligible and reads every digest again after restart. The local reference is not an OS sandbox, continuously scheduled worker deployment, production key provider, or production live-provider integration. Coordinated reference backup and isolated restore do not constitute provider-specific production disaster recovery. The exact baseline/candidate comparison reference now resolves exact retained upstream records in the default repository-backed composition and exposes immutable definitions, snapshots, and results through PostgreSQL, bounded HTTP/OpenAPI routes, the workspace TypeScript SDK, and a digest- and lineage-verifying operator view. Immutable release candidates and versioned release policy definitions now cross strict contracts, authority resolution, dedicated PostgreSQL roles, append-only storage, HTTP/OpenAPI, SDK, recovery, and clean-checkout restart boundaries. Policy source and reviewer records and the installation registry in the reference flow remain synthetic; publication does not evaluate a candidate or approve, enforce, or release anything. Explicitly requested policy evaluation now retains guarded sealed inputs, exact rule evidence, separate worker authority, bounded API/SDK, recovery and authenticated contributor checks. The SDK is not published to a package registry. Console sign-in integration, approval, decision, and release gates are intentionally not represented as complete.

Why ProofStack

Agent teams need more than logs and dashboards. They need inspectable evidence for five questions:

1. What did the agent do?
2. Why did it take that path?
3. Was the outcome correct, safe, and economical?
4. Can the execution be reproduced?
5. Should this version be allowed to run in production?

ProofStack is designed around a continuous reliability loop:

observe -> reproduce -> evaluate -> enforce -> release -> learn

The implemented Workflow 1 reference instruments a tool-using agent, inspects its linked evidence trace, turns a failure into an immutable regression fixture, performs exact recorded replay, evaluates retained evidence, and derives a descriptive baseline/candidate comparison. It does not block a release. Policy enforcement and accountable release decisions remain Workflow 2 work.

What works today

Surface	Foundation capability
Contract	Strict, versioned, provider-neutral EvidenceEnvelope with W3C trace identity
Core	Tenant-scoped authorization, idempotent ingestion, conflict detection, atomic batches
API	Health, direct JSON ingestion, exact trace and Workflow 1 record reads, bounded control-plane mutations, stable problem documents, OpenAPI 3.2
OTLP interoperability	OTLP 1.11 trace JSON/Protobuf, gzip, partial success, bounded normalization, and authenticated scope routing
Persistence	Checksum-verified PostgreSQL migrations, forced RLS, append-only evidence, atomic outbox
Delivery state	Leased outbox retries, poison-message visibility, monotonic cursors, consumer receipts
Workload identity	One-time API keys, bounded delegation, memory-hard hashes, rotation, revocation, audit, and isolated DB access
Human identity	OIDC Authorization Code + PKCE, explicit issuer/subject bindings, encrypted one-time transactions, authoritative revocable sessions, and CSRF defense
Artifact lifecycle	Opt-in classified metadata, envelope encryption, immutable S3-compatible objects, PostgreSQL tombstones and purge receipts
Artifact operations	Scoped reconciliation, retention, abandoned-upload cleanup, purge retry, and referenced-key inspection
Recovery	Fail-closed PostgreSQL dumps, canonical recovery manifests and inventories, empty-target coordinated restore, audited replay-epoch fencing, fresh roles, and tenant-adversarial verification
Regression catalog	Immutable observed trace snapshots and ordered dataset versions through memory, PostgreSQL, API, OpenAPI, SDK, outbox, and recovery boundaries
Interaction capture	Fixture-owned classified model and tool attempts, exact artifact lineage, metadata/content export, revocation, purge, and recovery
Recorded-boundary replay	Strict full-content preflight, ordered exact normalized-request matching, no live fallback, cooperative fixed runtime inputs, and bounded or unknown results
Durable replay jobs	Immutable releases and plans, finite multidimensional budgets, fenced leases and restore epochs, cancellation, predeclared retry/effect rules, usage reconciliation, separate worker/target processes, and durable result artifacts
Non-model evaluation primitives	Total tri-state applicability, digest-registered exact-byte and bounded JSON Schema oracles, exact five-verdict counts, and assumption-gated Wilson intervals
Evaluation graph boundary	Sixteen immutable record kinds, memory and PostgreSQL exact-scope repositories, authorization-first server authorship, RLS, lineage, idempotency, outbox, and recovery
Evaluation service entry	Exact-version API and fail-closed SDK, separate least-privilege evaluation-worker storage authority, five-verdict contested reference flow, and restart read-back
Model and human assurance	Thirteen strict record kinds, exact model/prompt/tool lineage, mandatory-slice qualification, calibration compatibility, blinded order swaps, independent critique, reviewer accountability, and conservative assessment
Assurance authority	Kind-routed control, model-worker, and human-review PostgreSQL roles backed by API capability checks, RLS, append-only lineage, recovery, and complete restart read-back
Exact evidence comparison	Repository-backed retained-source resolution, strict definitions and snapshots, exact case pairing and arithmetic, immutable memory/PostgreSQL repositories, HTTP/OpenAPI, workspace SDK, and a digest-verifying operator view
Release candidates	Exact checked-out commit/tree, runtime identity, retained Workflow 1 lineage, immutable memory/PostgreSQL records, API/SDK, restart read-back, and recovery
Release policy definitions	Finite non-executable rules, exact applicability and authority references, independent source-review lineage, dedicated author storage authority, immutable lifecycle, API/SDK, restart read-back, and recovery
TypeScript SDK	Generated IDs, bounded telemetry delivery, and fail-closed exact-version regression, replay, evaluation, model-assurance, comparison, release-candidate, release-policy and policy-evaluation control/read clients with explicit authentication modes
Console	API health, exact linked-trace inspection, and digest- and lineage-verified comparison reads without placeholder telemetry, classified plaintext, or release controls
Examples	Runnable trace, evidence-only regression, capture-to-recorded replay, durable success/cancellation/stale-fence recovery, contested evaluation/assurance, exact synthetic comparison, and disposable retained Workflow 1, release-candidate, and policy-publication acceptance paths
Engineering	Monorepo boundaries, strict TypeScript, coverage, production builds, pinned CI actions
Security	Explicit threat model, safe production startup refusal, dependency and secret scanning

The end-to-end foundation is deliberately dependency-light:

flowchart LR
    A[Observed agent] -->|EvidenceRecord| S[TypeScript SDK]
    S -->|bounded batch| H[Fastify API]
    T[OTLP exporter or collector] -->|OTLP/HTTP traces| H
    H -->|PrincipalContext| C[Core use cases]
    C -->|tenant-scoped port| R{Evidence repository}
    R --> M[(Memory quickstart)]
    R --> P[(PostgreSQL)]
    P --> X[(Transactional outbox)]
    H --> Q[(Replay definitions and jobs)]
    Q --> RW[Least-privilege replay worker]
    RW --> TP[Exact target process]
    W[Operator console] -->|validated response| H
    H --> O[OpenAPI contract]

Loading

The memory adapter keeps the quickstart dependency-free. The PostgreSQL adapter is the durable option: migration integrity, database-enforced tenant isolation, immutable evidence, atomic evidence/outbox writes, and ten isolated least-privilege runtime roles are covered by real PostgreSQL tests. The experimental API-key mode is end-to-end functional for workloads. The OIDC browser API is functional with server-side bindings and sessions; provider deployment validation and operator console sign-in integration remain unfinished. Artifact lifecycle and interaction-capture operations are available through the API and TypeScript SDK as well as domain libraries and one-shot operator commands. Continuous maintenance scheduling and a production external key provider remain unfinished. The bounded OTLP/HTTP trace profile accepts standard JSON or binary Protobuf exporters at /v1/traces; OTLP/gRPC, non-trace signals, distributed quotas, and a production collector matrix remain outside the implemented claim.

Quickstart

Requirements: Node.js 24+ and pnpm 11.24.0.

For the CI runtime baseline, use the exact Node.js version in .nvmrc (currently 24.19.0).

git clone https://github.com/Kwondh0321/proofstack.git
cd proofstack
pnpm install --frozen-lockfile
pnpm check
pnpm dev

The API starts at http://127.0.0.1:4318, its machine-readable contract at http://127.0.0.1:4318/openapi.json, and the console at http://127.0.0.1:3000.

With the API still running, send a real SDK trace from another terminal:

pnpm example:basic-agent

The evidence transport rejects HTTP redirects and omits ambient browser credentials. Configure the final ingestion endpoint directly; remote endpoints require HTTPS, while explicit loopback HTTP remains available for local development.

Then exercise the first Workflow 1 vertical path:

pnpm example:incident-to-regression

The second command emits a failed trace, freezes one exact evidence-only fixture, publishes one dataset version, reads both versions back, and prints their immutable digests. See the incident-to-regression guide for authority, idempotency, failure, and non-replay boundaries.

To execute the real exact comparison engine without starting a server or database:

pnpm example:comparison-control-flow

The default experiment freezes synthetic 125 ms baseline and 100 ms candidate evidence, derives an exact -25 ms delta, persists the immutable result, and reads it back. Change the input measurements or inspect the precise limitations in the comparison experiment guide.

To exercise the same comparison through the actual HTTP API and operator console, keep these two commands running in separate terminals, then open http://127.0.0.1:3011/comparisons/result_latency_service:

PROOFSTACK_WEB_PORT=3011 pnpm example:comparison-api
pnpm --filter @proofstack/web exec next dev --port 3011

Port 3011 is optional and only avoids a collision when the normal port 3000 is already occupied. The API-backed demonstration is synthetic and memory-only; the guide documents expected values, custom inputs, shutdown, and the production claims it deliberately does not make.

To capture and then revoke an exact provider-neutral model/tool interaction boundary:

pnpm example:interaction-capture

The example stores eleven dedicated classified artifacts, publishes an immutable recorded_interactions successor, verifies plaintext-free metadata and acknowledged exact-content exports, runs one exact recorded model/tool flow and one forced mismatch outside the API process, then tombstones and purges the complete owned set. See the interaction-capture guide, the recorded-boundary replay guide, and the local development guide for authority, failure behavior, configuration, and troubleshooting.

The durable replay reference needs the PostgreSQL and S3-compatible profiles and therefore has a separate setup procedure. It publishes an exact target release and plan, runs success, cancellation, and expired-lease recovery jobs through separate worker and target processes, and persists each terminal report before success. Follow the durable replay guide; its local credentials and bounded process profile must not be treated as production configuration or isolation.

The model-assisted and human-evaluation reference is a self-contained PostgreSQL integration harness. It provisions disjoint control, model-worker, and human-review roles; preserves injection, calibration, order, independence, counterevidence, provider-failure, and dissent signals; restarts the API; and re-reads every digest. Follow the model-assisted and human evaluation guide. It uses a deterministic local provider and synthetic reviewers, not a live model or production identity deployment.

To run the complete retained Workflow 1 graph once in disposable infrastructure, follow the Workflow 1 acceptance guide. Its single root command chooses random loopback ports and removes its isolated PostgreSQL and object-storage volumes on success or failure; no long-lived site or fixed application port is required.

Repository map

apps/api                 HTTP composition root and development ingestion API
apps/web                 Server-rendered operator console
packages/contracts       Runtime schemas, public types, identity, and OpenAPI generation
packages/core            Framework-independent authorization and evidence use cases
packages/source-authority  Scoped local Git and declared candidate reference authority adapters
packages/datasets        Immutable regression definitions, binary encoding, and public vectors
packages/replay          Canonical bounded-replay definitions and fail-closed recorded execution
packages/policy-evaluation  Bounded cross-domain record graph acquisition; not a release decision
packages/artifacts       Encrypted content lifecycle, authorization, and storage ports
packages/postgres        Durable repositories, migrations, delivery state, and runtime roles
packages/recovery        Coordinated recovery manifests, object inventories, and verification
packages/s3              Immutable S3-compatible artifact object adapter
services/artifact-maintenance  Scoped one-shot lifecycle and key-safety commands
services/recovery        Safe logical database operations and isolated recovery rehearsal
services/replay-worker   Fenced durable-attempt execution, accounting, and boundary supervision
services/evaluation-worker  Least-privilege non-model evaluation evidence recorder
services/model-evaluation-worker  Least-privilege model execution evidence recorder
sdks/typescript          Provider-neutral telemetry and exact Workflow 1 control-plane clients
examples/basic-agent     Verified SDK-to-API trace example
examples/incident-to-regression  Executable evidence-only regression catalog flow
examples/interaction-capture  Provider-neutral capture, recorded replay, mismatch, and revocation flow
examples/durable-replay  Durable success, cancellation, stale-fence recovery, and result flow
examples/evaluation-control-flow  Contested non-model service and restart flow
examples/model-assurance-control-flow  Adversarial model/human assurance and recovery flow
examples/comparison-control-flow  Exact synthetic baseline/candidate engine experiment
docs/architecture        Numbered architecture decision records
docs/product             Product constitution and dependency-ordered roadmap
docs/operations          Deployment contracts and operator procedures
docs/security            Trust boundaries, threats, controls, and production gates
scripts                  Repository-level architecture enforcement

Internal dependency direction is enforced during pnpm check; applications cannot silently leak framework or storage concerns into contracts and core logic.

Non-negotiable invariants

• Tenant ownership is assigned from server-authenticated context, never a client payload.
• Received evidence is immutable and idempotent; conflicting reuse of an event ID is rejected.
• Telemetry failure does not crash the observed workload by default.
• Mandatory policy enforcement will fail closed when that capability is implemented.
• Sensitive content capture is opt-in and separate from metadata-first evidence.
• Experimental or planned functionality is labeled honestly.

Read the product constitution before broad changes. Consequential technical decisions are recorded in ADRs, and capability order and acceptance gates live in the roadmap.

The cross-layer Foundation 1 audit records the findings closed before durable storage work and the limitations that still block production use. The Foundation 2 durable core audit records the accepted PostgreSQL and delivery-state checkpoint without claiming the remaining stage is done. The Foundation 2 identity audit records the accepted workload and browser identity checkpoint and its remaining deployment limitations. The Foundation 2 artifact audit records the accepted encrypted lifecycle and operator checkpoint without claiming API or production-key composition. The Foundation 2 OTLP/HTTP audit records the accepted trace interoperability checkpoint, independent exporter evidence, and remaining production gaps. The Foundation 2 recovery and isolation audit records coordinated empty-target restoration, migration and tenant barriers, the stage exit, and the limits that still block a production-readiness claim. The Workflow 1 regression catalog audit accepts only the immutable evidence-only catalog checkpoint and lists the replay, evaluation, and comparison work that remains open. The Workflow 1 interaction-capture audit accepts classified, fixture-owned interaction evidence while explicitly withholding executable replay authority. The recorded-boundary replay entry audit defines the checkpoint's entry gates. The completed recorded-boundary replay audit accepts exact recorded matching with explicit same-process limits while withholding durable-job, evaluation, and release authority. The durable replay-job entry audit defines the checkpoint's release, budget, fencing, cancellation, worker, persistence, recovery, and authority gates. The completed durable replay-job audit accepts the bounded execution boundary against green local and service gates while withholding evaluation, approval, release, and production-readiness claims. The non-model evaluation primitives guide documents the core-only applicability, oracle, and aggregate boundary that preceded the accepted service-backed checkpoint. The service-backed evaluation control-flow guide runs the exact API, SDK, worker-role, PostgreSQL, and restart path while preserving stale sources, counterevidence, disagreement, low coverage, and an ineligible conclusion. The completed criteria and non-model evaluation audit accepts that immutable evidence and eligibility boundary while withholding model-assisted evaluation, comparison, policy, approval, release, and production-readiness claims. The model-assisted and human evaluation guide documents the bounded local-provider, model-worker, human-review, authority, failure, and restart flow. The completed model-assisted and human evaluation audit accepts that contestable assurance checkpoint while withholding baseline/candidate product comparison, policy, approval, release, live-provider, and production-readiness claims. The completed baseline/candidate comparison audit accepts the immutable descriptive comparison boundary while withholding policy, approval, release, causal, and production-readiness claims. The completed Workflow 1 end-to-end audit accepts the bounded reference incident-to-comparison stage and opens Workflow 2 development while withholding policy, approval, deployment, release, and production-readiness claims. The Workflow 2 entry audit fixes the authority split and seven dependency-ordered gates. The immutable release-candidate guide runs the first bounded subject slice against the complete retained Workflow 1 graph while withholding policy, approval, decision, attestation, CI-enforcement, deployment, and production-readiness claims. The completed release-candidate audit accepts that first bounded checkpoint and opens versioned policy-definition development. The versioned policy-definition entry audit fixes the finite non-executable rule vocabulary, installation and source authority, immutable lifecycle, dedicated author boundary, and executable exit gates without claiming evaluation or release authority. The immutable release-policy guide runs the retained source, reviewer, installation, publication, withdrawal, and restart path while withholding policy-evaluation, approval, decision, attestation, CI-enforcement, deployment, and production-readiness claims. The completed policy-definition audit records the second Workflow 2 checkpoint's acceptance, closed cross-check findings, exact-commit publication gate, and retained limits. Definition acceptance does not establish that a policy is correct, satisfied, or authorized for a release; subsequent evaluation has its own completion gate. The policy-evaluation entry audit fixes sealed input selection, exact arithmetic, missingness, declarative approval prerequisites, separate worker authority, durable execution, and acceptance gates. It approves implementation entry only. The later policy-evaluation audit accepts the bounded third checkpoint subject to its exact-commit publication gate; accountable decisions and approvals remain next. The historical prerequisite notes below retain their original scope and do not replace that audit or the current roadmap. The comparison-selection building block now accounts for the candidate's complete declared comparison set and preserves unique, missing, ambiguous, and unresolved mappings without claiming complete lineage validation or a sealed input. The comparison-lineage building block then validates the exact definition and snapshots, re-derives the selected result, binds the candidate-owned dataset, target, and assurance references, and emits a bounded direct-source frontier without claiming that subordinate records are already verified or sealed.

Current boundaries

The current build does not provide console-integrated OIDC sign-in, a production external artifact key provider, continuously scheduled artifact workers, OTLP/gRPC or non-trace signal ingestion, a deployed outbox publisher, a continuously scheduled production replay-worker deployment, OS/container-isolated target or evaluator workers, production live-provider model evaluation, policy enforcement, continuous provider-specific disaster recovery, or production deployment artifacts. Immutable evidence-only regression versions, fixture-owned classified interaction capture, recorded-boundary replay, and bounded durable replay jobs with separate local processes are implemented and tested, alongside workload API-key and OIDC browser authentication, artifact lifecycle, and the OTLP/HTTP trace profile. Non-model evaluation primitives and model/human assurance contracts, the authorization-first immutable graphs, durable PostgreSQL adapters, exact-version API and SDK, kind-routed storage authorities, dedicated workers, and restart read-back are implemented and tested. Repository-backed exact comparison source resolution, immutable comparison records, bounded API and OpenAPI operations, workspace SDK methods, and the read-only operator projection are also implemented and tested. Strict release-candidate contracts, exact source resolution, append-only memory and PostgreSQL storage, bounded API and SDK operations, restart read-back, and empty-target predecessor recovery are implemented and tested as a policy-independent Workflow 2 subject. Versioned release-policy contracts, finite non-executable rules, exact installation and qualified source authority, append-only memory and PostgreSQL storage, a dedicated policy-author role, bounded HTTP/OpenAPI and SDK operations, immutable withdrawal history, restart read-back, recovery, and clean-checkout acceptance are implemented and tested as a definition-only Workflow 2 boundary. The acceptance completion correction records the earlier mode-forwarding limitation and requires exact requested-mode evidence. The authenticated policy contributor connects the retained graph, scoped API-key SDK and separate worker. Its requirement-by-requirement exit evidence is recorded in the policy-evaluation audit above. See the evaluation repository and use cases guide. The reference flows record synthetic evidence and use a deterministic local model provider; ProofStack does not yet execute arbitrary evaluators in an OS sandbox, determine source authority automatically, validate real reviewer expertise, publish its workspace SDK to a package registry, automatically select a release policy for a candidate, or reinterpret a descriptive comparison as a release decision. Replay does not claim OS-enforced network, filesystem, process, or dependency isolation. The built-in content inspector rejects structured credential fields and supports configured scanners, but no scanner proves arbitrary opaque bytes secret-free; scanner qualification, distributed quotas, and a production exporter/collector matrix remain deployment-owned. Foundation 2's coordinated recovery reference is implemented and tested against pinned CI services, but external key recovery, immutable provider backups, measured RPO/RTO, off-site retention, and repeated deployment rehearsals remain operator-owned. Remaining capabilities have an explicit dependency order and may not bypass the security and compatibility gates described in the roadmap.

Contributing and security

Run pnpm check before every reviewable change and keep commits focused on one coherent decision. See CONTRIBUTING.md for the complete workflow.

Do not report vulnerabilities in a public issue. Follow SECURITY.md and review the foundation threat model.

License

Apache License 2.0. See LICENSE.