Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 35 additions & 25 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,27 +23,31 @@ jobs:
name: Typecheck
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: "20"
- uses: pnpm/action-setup@v4
with:
version: 9
node-version: "22"
# No `version` input: pnpm/action-setup errors out when the version is given both
# here and as `packageManager` in package.json, which is the single source of truth.
- uses: pnpm/action-setup@v6
- run: pnpm install --frozen-lockfile
# Cross-package types resolve through each package's built dist/*.d.ts (no project
# references, no path mapping), and dist/ is gitignored — so a clean checkout must
# build before it can typecheck, or every @pi-managed/* import fails with TS2307.
- run: pnpm build
- run: pnpm typecheck

lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"
- uses: pnpm/action-setup@v4
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
version: 9
node-version: "22"
# No `version` input: pnpm/action-setup errors out when the version is given both
# here and as `packageManager` in package.json, which is the single source of truth.
- uses: pnpm/action-setup@v6
- run: pnpm install --frozen-lockfile
- run: pnpm lint

Expand All @@ -63,18 +67,21 @@ jobs:
DOCKER_HOST: unix:///var/run/docker.sock
TESTCONTAINERS_RYUK_DISABLED: "true"
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: "20"
- uses: pnpm/action-setup@v4
with:
version: 9
node-version: "22"
# No `version` input: pnpm/action-setup errors out when the version is given both
# here and as `packageManager` in package.json, which is the single source of truth.
- uses: pnpm/action-setup@v6
- name: Verify container runtime (fail-closed prerequisite)
run: docker info
- name: Pre-pull test images
run: docker pull postgres:16-alpine
- run: pnpm install --frozen-lockfile
# Same reason as the typecheck job: suites import @pi-managed/* by package name,
# which resolves to the gitignored dist/.
- run: pnpm build
- run: pnpm test

# @kvm tests: the real microsandbox provider, the sandbox-operations adapter, the
Expand All @@ -88,7 +95,7 @@ jobs:
# goes green) rather than passing vacuously — that is the intended fail-closed shape.
# * Linux host — bare metal, or a VM with NESTED VIRTUALIZATION enabled — exposing
# /dev/kvm, with the runner's user in the `kvm` group (/dev/kvm must be writable).
# * Node.js >= 20 + pnpm on PATH; a container runtime (docker or podman) for the
# * Node.js >= 22.19 + pnpm on PATH; a container runtime (docker or podman) for the
# Postgres-backed halves of the E2E and credential-injection gates.
# * microsandbox one-time bootstrap:
# node -e "import('microsandbox').then(m => m.install())"
Expand All @@ -108,19 +115,22 @@ jobs:
PI_REQUIRE_INTEGRATION: "1"
TESTCONTAINERS_RYUK_DISABLED: "true"
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"
- uses: pnpm/action-setup@v4
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
version: 9
node-version: "22"
# No `version` input: pnpm/action-setup errors out when the version is given both
# here and as `packageManager` in package.json, which is the single source of truth.
- uses: pnpm/action-setup@v6
- name: Verify /dev/kvm
run: |
test -e /dev/kvm || { echo "::error::/dev/kvm missing — this runner is not KVM-capable"; exit 1; }
test -w /dev/kvm || { echo "::error::/dev/kvm not writable by the runner user (add it to the 'kvm' group)"; exit 1; }
ls -l /dev/kvm
- run: pnpm install --frozen-lockfile
# Same reason as the typecheck job: the @kvm suites import @pi-managed/* by package
# name, which resolves to the gitignored dist/.
- run: pnpm build
- name: Verify msb native package installed
run: ls node_modules/@superradcompany/
- name: Bootstrap microsandbox runtime (idempotent)
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/db-schema.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,13 @@ jobs:
name: docs/db-schema.md matches migrations
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: "20"
- uses: pnpm/action-setup@v4
with:
version: 9
node-version: "22"
# No `version` input: pnpm/action-setup errors out when the version is given both
# here and as `packageManager` in package.json, which is the single source of truth.
- uses: pnpm/action-setup@v6
- run: pnpm install --frozen-lockfile
# The generator boots postgres:16-alpine itself via the container CLI (docker on the
# GitHub runner, podman locally) and runs psql inside it — no host pg client needed.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ The service is **tailored for Pi** — it adapts the *concepts* of managed agent

### Prerequisites

- **Node.js ≥ 20**
- **Node.js ≥ 22.19** (the embedded Pi agent's `undici` requires it)
- **pnpm** (`corepack enable` — the version is pinned via `packageManager` in `package.json`)
- **Docker** (for the dev Postgres; the object store defaults to the local filesystem)
- **Linux with `/dev/kvm`** (for real microVM sandboxes — macOS works for everything *except* sandbox execution)
Expand Down
7 changes: 7 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,13 @@ Two deliberate wrinkles: the `testkit`↔`backend` relationship looks circular b
acyclic (types one way, devDependency the other), and the worker's isolation means the
work-item shape exists in two places that must be kept in sync by hand.

That first wrinkle has one concrete consequence: **`pnpm -r build` cannot order the
workspace on its own.** pnpm sees the cycle and may build `testkit` before `backend`,
which fails on a clean tree because cross-package types resolve only through each
package's built (gitignored) `dist/*.d.ts` — there are no project references and no path
mapping. The root `build` script therefore builds everything *except* `testkit` first,
then `testkit`. Run `pnpm build` — not `pnpm -r build` — after a fresh clone.

### 3.3 Deployment shapes

1. **Single-host self-hosted (v1 default):** one process, one implicit tenant, local
Expand Down
4 changes: 2 additions & 2 deletions docs/deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ runtime** — plus a Linux host with **KVM** for microVM isolation.
| Requirement | Why | Notes |
|---|---|---|
| **Linux host with `/dev/kvm`** | microsandbox microVMs run via libkrunfw + KVM | The sandbox provider is inert without it. `@kvm`-gated tests skip cleanly when absent. |
| **Node.js ≥ 20** | ESM service, `dist/main.js` entrypoint | Build with `pnpm --filter @pi-managed/backend build`. |
| **Node.js ≥ 22.19** | ESM service, `dist/main.js` entrypoint | The floor is set by the embedded Pi agent's `undici` (`engines: >=22.19.0`), not by our own code. Build with `pnpm --filter @pi-managed/backend build`. |
| **Postgres 16** | Control-plane DB (sessions, agents, environments, vaults, usage, …) | Migrations run **forward-only on boot** (`runMigrations(dbUrl, "up")`). |
| **Object store** | File payloads, memory stores, snapshots, JSONL sync (§28) | **Local filesystem by default** (`OBJECT_STORE_ROOT` — point it at durable storage), or **Google Cloud Storage** via `OBJECT_STORE_KIND=gcs` + `GCS_BUCKET` (§2). An S3-compatible adapter also exists (`infra/objectstore/s3.ts`) but is composition-time only. |
| **microsandbox runtime** | Provisiones/execs detached microVMs (§5.4, §10) | One-time `microsandbox` install bootstrap (§3 below). |
Expand Down Expand Up @@ -252,7 +252,7 @@ Runner prerequisites (infra work, outside this repo):
intended fail-closed behavior (a placeholder that echoes and exits 0 is not).
2. **Hardware virtualization**: bare metal, or a VM with nested virtualization enabled.
`/dev/kvm` must exist and be writable by the runner user (add it to the `kvm` group).
3. **Node ≥ 20 + pnpm** on `PATH`, and a **container runtime** (docker or podman) for the
3. **Node ≥ 22.19 + pnpm** on `PATH`, and a **container runtime** (docker or podman) for the
Postgres-backed halves of the E2E and credential-injection gates.
4. **microsandbox bootstrap**: `node -e "import('microsandbox').then(m => m.install())"`
(§3) populating `~/.microsandbox/`. The workflow re-runs this per job (idempotent), so
Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ path over production hardening. When you are ready to deploy for real, follow

| Tool | Why | Check |
|---|---|---|
| **Node.js ≥ 20** | The service is an ESM Node process | `node --version` |
| **Node.js ≥ 22.19** | The service is an ESM Node process; the embedded Pi agent's `undici` requires 22.19+ | `node --version` |
| **pnpm** | Workspace package manager (pinned via `packageManager`) | `corepack enable` then `pnpm --version` |
| **Docker** | Runs the dev Postgres | `docker info` |
| **Linux + `/dev/kvm`** | *Only* for real microVM sandboxes — optional to start | `ls -l /dev/kvm` |
Expand Down
2 changes: 1 addition & 1 deletion docs/spec/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -450,7 +450,7 @@ the start so neither shape is a retrofit.
virtualization requires an Intel x86 machine type (Haswell+) with
`enableNestedVirtualization` set — unavailable on ARM (T2A) — and carries a measurable
performance tax vs bare metal.
- **Runtime:** Node 20+.
- **Runtime:** Node 22.19+ (floor set by the embedded Pi agent's `undici`).
- **Dependencies:** Postgres (metadata, quotas, job records, webhook deliveries), an
object store (filesystem in v1; S3-compatible or GCS for SaaS), the microsandbox runtime
(`~/.microsandbox/` with libkrunfw + agentd).
Expand Down
2 changes: 1 addition & 1 deletion docs/user-journeys.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ all three roles — the layering still holds, they just switch hats (and keys).

*Goal: a working backend on one Linux/KVM machine.*

1. Provision a Linux host with `/dev/kvm`, Node ≥ 20, and Docker (or a managed
1. Provision a Linux host with `/dev/kvm`, Node ≥ 22.19, and Docker (or a managed
Postgres 16). The object store is a **local directory** by default — put it on
durable disk.
2. `pnpm install && pnpm build`; start the dependency (`docker compose up -d postgres`
Expand Down
6 changes: 3 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,19 @@
"description": "Pi Managed Backend — a control plane around the Pi coding agent with microsandbox microVM execution.",
"type": "module",
"engines": {
"node": ">=20"
"node": ">=22.19.0"
},
"packageManager": "[email protected]",
"scripts": {
"build": "pnpm -r build",
"build": "pnpm --filter '!@pi-managed/testkit' build && pnpm --filter @pi-managed/testkit build",
"test": "pnpm -r test",
"lint": "eslint .",
"typecheck": "pnpm -r typecheck",
"db:schema:gen": "node scripts/gen-db-schema.mjs",
"db:schema:check": "node scripts/gen-db-schema.mjs --check"
},
"devDependencies": {
"@types/node": "^26.1.1",
"@types/node": "^22.20.1",
"eslint": "^10.7.0",
"eslint-config-prettier": "^10.1.8",
"prettier": "^3.9.5",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -105,17 +105,19 @@ describe("ledger (integration)", () => {

const row = await getBillingRow(pool, tenantId);
expect(row?.balanceMicros).toBe(1_000_000);
// 41 entries: seed + 40 ops. The last entry's balance_after equals the balance.
const { rows } = await query<{ count: string; maxbal: string }>(
// 41 entries: seed + 40 ops, each recorded exactly once.
//
// We deliberately do NOT assert that the last entry's balance_after equals the
// balance: nothing here can identify the last-APPLIED entry. `created_at` defaults
// to `now()`, which is transaction-START time in Postgres, so it does not track the
// order in which concurrent transactions take the `FOR UPDATE` lock, and `id` is
// random. The authoritative no-lost-update guarantee is `balanceMicros` above.
const { rows } = await query<{ count: string }>(
pool,
`SELECT count(*)::text AS count,
(SELECT balance_after_micros FROM ledger_entries
WHERE tenant_id = $1 ORDER BY created_at DESC, id DESC LIMIT 1)::text AS maxbal
FROM ledger_entries WHERE tenant_id = $1`,
"SELECT count(*)::text AS count FROM ledger_entries WHERE tenant_id = $1",
[tenantId],
);
expect(rows[0].count).toBe("41");
expect(rows[0].maxbal).toBe("1000000");
});

it("concurrent replays of the SAME key apply exactly once", async () => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -195,5 +195,9 @@ describe.skipIf(!RUNTIME)("scheduler batching (§17.8)", () => {
// instant re-enumerates from MAX(scheduled_at) and records nothing new.
await scheduler.tick();
expect(await runsFor(jobId)).toHaveLength(minutes);
}, 60_000); // enumerating > one chunk of minute occurrences is the slow part
// Enumerating > one chunk of minute occurrences is the slow part: 5,200 of them,
// twice, against a containerized Postgres. 60s was enough on developer hardware but
// not on a 2-core CI runner sharing the box with the other parallel suites, so this
// matches the package's 180s hookTimeout rather than sitting just under the wire.
}, 180_000);
});
6 changes: 4 additions & 2 deletions packages/backend/src/domain/session-manager/jsonl-sync.ts
Original file line number Diff line number Diff line change
Expand Up @@ -202,8 +202,10 @@ function fileStream(path: string): ReadableStream<Uint8Array> {
const nodeStream = createReadStream(path);
return new ReadableStream<Uint8Array>({
start(controller) {
nodeStream.on("data", (chunk: Buffer) => {
controller.enqueue(new Uint8Array(chunk));
// No encoding is set on the stream above, so chunks are always Buffer; `data`
// widens to string only for streams that set one.
nodeStream.on("data", (chunk: string | Buffer) => {
controller.enqueue(new Uint8Array(chunk as Buffer));
});
nodeStream.on("end", () => controller.close());
nodeStream.on("error", (err) => controller.error(err));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -264,7 +264,10 @@ d("session-worker process pool (R7.1)", () => {
expect(rtB).toBeDefined();
await rtB.sendEvent(userMessage("still alive?"));
expect(await managed.eventsStore.head(sessionB)).toBeGreaterThan(headBBefore);
expect(rtB.status()).toBe("idle");
// `sendEvent` resolves once the event is accepted; the turn settles back to idle
// asynchronously after that, so poll rather than sampling once. A session that
// never leaves `running` still fails the assertion when `waitFor` times out.
expect(await waitFor(() => rtB.status() === "idle")).toBe(true);
}, 60_000);

it("(c) a killed child's session recovers (re-attach) on the next getOrCreate", async () => {
Expand Down
10 changes: 9 additions & 1 deletion packages/backend/src/infra/db/pool.ts
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,15 @@ export interface TenantCtx {
* {@link closePool}.
*/
export function createPool(config: PoolConfig): Pool {
return new pg.Pool(config);
const pool = new pg.Pool(config);
// `pg` emits `error` on IDLE pooled clients when the server drops them —
// a restart, a failover, or `pg_terminate_backend` (`57P01`). An `error` event with no
// listener is fatal to the Node process, so without this a routine Postgres restart
// takes the backend down with it. The pool discards the dead client on its own and
// stays usable, and an error on a CHECKED-OUT client still rejects that client's own
// query, so nothing is being swallowed that a caller would otherwise see.
pool.on("error", () => {});
return pool;
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -393,7 +393,10 @@ function runContract(
const put = await store.put(key, byteStream(payload));
const got = await readAll(await store.get(key));
expect(got.byteLength).toBe(payload.byteLength);
expect(got).toEqual(payload);
// Same byte-exact guarantee as the small-payload case, but native: `toEqual`
// walks a multi-MiB typed array element by element, which timed out the 5s
// default on a shared CI runner (~2.4s per store even on fast hardware).
expect(Buffer.compare(Buffer.from(got), Buffer.from(payload))).toBe(0);
// ETag must be stable across reads of the same content: a conditional put
// with the returned etag must succeed.
const cp = await store.conditionalPut(
Expand Down
2 changes: 1 addition & 1 deletion packages/billing-adapter/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
"@pi-managed/backend": "workspace:*",
"@pi-managed/contracts": "workspace:*",
"@testcontainers/postgresql": "^10.18.1",
"@types/node": "^26.1.1",
"@types/node": "^22.20.1",
"pino": "^10.1.0",
"typescript": "^5.9.3",
"vitest": "^4.1.10"
Expand Down
5 changes: 4 additions & 1 deletion packages/testkit/src/conformance/object-store.ts
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,10 @@ export function runObjectStoreConformance(
const put = await store.put(key, byteStream(payload));
const got = await readAll(await store.get(key));
expect(got.byteLength).toBe(payload.byteLength);
expect(got).toEqual(payload);
// Same byte-exact guarantee as the small-payload case, but native: `toEqual`
// walks a multi-MiB typed array element by element, which cost ~2.4s of this
// suite's ~2.5s and timed out the 5s default on a shared CI runner.
expect(Buffer.compare(Buffer.from(got), Buffer.from(payload))).toBe(0);
// ETag must be stable across reads of the same content: a conditional put
// with the returned etag must succeed.
const cp = await store.conditionalPut(
Expand Down
2 changes: 1 addition & 1 deletion packages/worker/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
},
"dependencies": {},
"devDependencies": {
"@types/node": "^26.1.1",
"@types/node": "^22.20.1",
"typescript": "^5.9.3",
"vitest": "^4.1.10"
}
Expand Down
4 changes: 4 additions & 0 deletions packages/worker/src/control-levels.ts
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,10 @@ export class SpawnExecutor implements Executor {
}
});

// The script may exit before reading stdin, closing the pipe mid-write. `close`
// already reports that exit as an error result, so the EPIPE has nothing to add —
// but with no listener it surfaces as an unhandled error and takes the process down.
child.stdin.on("error", () => {});
child.stdin.end(payload);
});
}
Expand Down
Loading
Loading