diff --git a/README.md b/README.md index c7b6721..0a3dea5 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ organizing principle: | Package | Role | | ------- | ---- | -| [`@mieweb/cloud-types`](packages/cloud-types) | The portable contracts (`CloudDatabase`, `CloudBucket`, `CloudKV`, `CloudQueue`, `CloudStatefulNamespace`, `CloudVectorIndex`, `CloudAI`) plus `UnsupportedBindingError`. On Cloudflare these are exact aliases of the native binding types. | +| [`@mieweb/cloud-types`](packages/cloud-types) | The portable contracts (`CloudDatabase`, `CloudBucket`, `CloudKV`, `CloudQueue`, `CloudStatefulNamespace`, `CloudVectorIndex`, `CloudAI`, `CloudContainerNamespace`) plus `UnsupportedBindingError`. On Cloudflare these are exact aliases of the native binding types. | | [`@mieweb/cloud-workers`](packages/cloud-workers) | The `DurableObject` base. Backs the **`mieweb:workers`** virtual import: re-exports `cloudflare:workers` on Cloudflare (workerd export condition), pure-JS base everywhere else. | | [`@mieweb/cloud`](packages/cloud) | Umbrella entry — re-exports the contracts + `DurableObject` from one stable import surface. | | [`@mieweb/cloud-local`](packages/cloud-local) | Local/Node **adapters**: D1→SQLite, R2→filesystem, KV→in-memory, Queues→in-process, Durable Objects→in-process registry. Vectorize/Workers AI surface explicit `UnsupportedBindingError`. Includes a Node host harness + migration runner. | @@ -98,6 +98,24 @@ failing (e.g. Workers AI needs a model backend; Cloudflare local dev can't reach Vectorize/AI without credentials), so one suite stays green everywhere. The `local` target also runs as a plain `node --test` under `pnpm -r test`. +## Containers + +Cloudflare **Containers** (DO-controlled Linux containers) are a reserved +surface of the contract: `CloudContainerNamespace` / `CloudContainerStub` in +`@mieweb/cloud-types`. App code uses the stock Cloudflare shape — a +`class MyContainer extends Container` (from `@cloudflare/containers`) paired +with a `containers` entry + DO binding in `wrangler.jsonc` — and stays +target-agnostic. + +| Target | Support | +| ------ | ------- | +| `cloudflare` | ✅ native (`wrangler dev` / `wrangler deploy`, `wrangler containers ssh` for a shell) | +| `local` | ⏳ planned — Docker-backed adapter; today a container binding throws `UnsupportedBindingError` on use | +| `mieweb` | ⏳ planned — image distribution via skopeo → Harbor, cluster runtime TBD | + +See [container-plan.md](container-plan.md) for the milestones, the skopeo/Harbor +image pipeline, and the `myapp` walkthrough. + ## Status Proof-of-concept. Consumed today as a git submodule; npm / Deno / Bun diff --git a/container-plan.md b/container-plan.md new file mode 100644 index 0000000..6df5d69 --- /dev/null +++ b/container-plan.md @@ -0,0 +1,332 @@ +# Containers surface — plan + +Add Cloudflare **Containers** as a portable surface of the `@mieweb/cloud` contract, +cheap parts first. A Container on Cloudflare is a Durable-Object-controlled Linux +container (`class MyContainer extends Container`, bound via the normal +`durable_objects.bindings` block plus a `containers` array in `wrangler.jsonc`), so +the contract addition is small — the real work is image distribution and the +non-Cloudflare runtime, which we defer. + +**Key choices** + +- **skopeo for image movement.** No Docker daemon needed to *distribute*: the CLI + builds once (buildah/docker, whichever is present) and uses + `skopeo copy` / `skopeo inspect` to push the OCI image to each target's registry + (Cloudflare's managed registry via `wrangler containers push`, Harbor for the + `mieweb` target, `oci-archive:`/`containers-storage:` for local caching). +- **Harbor + Forgejo live in the opensource-server standalone cluster.** The + `mieweb` target's registry endpoint is Harbor; Forgejo Actions is the eventual + CI that builds + skopeo-copies images. This plan only reserves the config + surface for that (registry URL + creds in `mieweb.jsonc` targets block); the + cluster deploy itself is tracked in opensource-server, not here. +- **Fail-loudly first.** Until a real adapter exists, non-Cloudflare targets + surface `UnsupportedBindingError` through the existing + `createUnsupportedBinding` proxy — same pattern as Vectorize/AI on `local`. + +--- + +## Use case: `myapp` — a doc-conversion app that needs a container + +The story every doc/README anchors to. A dev is building **myapp**: users upload +`.docx` files, myapp serves them back as PDFs. The upload/list/serve logic is a +perfect Worker (R2 + D1 + KV), but the conversion needs **LibreOffice** — a +multi-GB Linux userland that can never run in a V8 isolate on *any* of our +targets. That's the container. + +**Shape of myapp:** + +``` +myapp/ +├── wrangler.jsonc # bindings: DOCS (R2), DB (D1), CONVERTER (container) +├── mieweb.jsonc # targets: local, mieweb (Harbor registry block) +├── worker/index.mjs # fetch handler + `class Converter extends Container` +└── converter/ + ├── Dockerfile # FROM debian + libreoffice + a tiny HTTP shim on :4000 + └── serve.mjs # POST /convert (docx in → pdf out) +``` + +The worker stays the brains; the container is a dumb HTTP appliance: + +```js +// worker/index.mjs +import { Container, getContainer } from '@cloudflare/containers'; + +export class Converter extends Container { + defaultPort = 4000; + sleepAfter = '5m'; // LibreOffice is heavy — sleep when idle +} + +export default { + async fetch(req, env) { + const url = new URL(req.url); + if (url.pathname === '/convert') { + const docx = await env.DOCS.get(url.searchParams.get('key')); + // one container instance per user keeps sessions warm but isolated + const converter = getContainer(env.CONVERTER, url.searchParams.get('user')); + const pdf = await converter.fetch('http://converter/convert', { + method: 'POST', body: docx.body, + }); + await env.DOCS.put(`pdf/${url.searchParams.get('key')}.pdf`, pdf.body); + return new Response('converted'); + } + // …upload/list/serve routes: plain R2/D1, already portable today + }, +}; +``` + +**The dev's day, per milestone:** + +| Dev action | Today | After M0–1 | After M2–3 | After M4 | +| --- | --- | --- | --- | --- | +| Write the code above | types don't exist | ✅ compiles against `CloudContainerNamespace` | ✅ | ✅ | +| `mieweb --target cloudflare dev` / `deploy` | ✅ works already (wrangler native) | ✅ | ✅ | ✅ | +| `mieweb --target local dev` — worker routes | ✅ | ✅ | ✅ | ✅ | +| `mieweb --target local dev` — hit `/convert` | crash at boot | clean `UnsupportedBindingError: CONVERTER on "local"` | same | ✅ container runs via local Docker | +| `mieweb images push --target mieweb` | — | — | ✅ buildah → skopeo → Harbor, digest locked | ✅ | +| `mieweb --target mieweb deploy` | — | — | image is in Harbor, runtime pending | ✅ end-to-end on the cluster | + +The value proposition to sell in docs: **myapp's repo never mentions a target.** +The same `worker/` + `converter/Dockerfile` pair deploys to Cloudflare's edge or +the opensource-server cluster; only `mieweb.jsonc` (registry pointer) differs — +and CI (Forgejo Actions) does the build + skopeo copy so the dev usually never +runs `images push` by hand. + +--- + +## Milestone 0 — Contract types (cheap, land now) + +Reserve the surface in `@mieweb/cloud-types` so app code can be written against it. + +- [x] Add `CloudContainerNamespace = DurableObjectNamespace` alias in + `packages/cloud-types/src/index.ts` (a Container binding *is* a DO namespace). +- [x] Add `CloudContainerStub = DurableObjectStub` alias (what `getContainer()` returns). +- [x] Document the row in the primitive table in the file header: + `Containers | DO-controlled Linux container | CloudContainerNamespace`. +- [x] Note in the doc comment that the app-side base class comes from + `@cloudflare/containers` (`Container`) and is Cloudflare-shaped; portable + adapters must emulate `ctx.container` (Milestone 4, deferred). +- [x] Update `packages/README.md` + root `README.md` binding tables with the new + row, marked *cloudflare-only for now* (like Vectorize/AI were at POC time). + +## Milestone 1 — Config schema + CLI awareness (cheap) + +Teach the sidecar config and CLI that containers exist, without implementing them. + +- [x] `packages/cli/mieweb-config.schema.json`: add `"docker"` (registry + hint) and keep `"unsupported"` as valid drivers for a container binding; add + per-target `registry` object: `{ url, project, username, password | authFile, + insecureSkipTlsVerify? }` (Harbor for `mieweb`). +- [x] CLI (`packages/cli/src/config.mjs`): parse the `containers` array from + `wrangler.jsonc` alongside `durable_objects.bindings` so container-backed DO + bindings are identifiable (binding name → class name → image). +- [x] `mieweb --target cloudflare deploy/dev`: no behavior change — wrangler + already handles `containers` natively; just make sure the CLI passes the + config through untouched. +- [x] `mieweb --target local|mieweb dev`: on encountering a container-backed DO + binding, wire `createUnsupportedBinding(name, target)` with a hint pointing + at this plan ("container adapter not yet implemented"). +- [x] Sample config: extend `packages/cloud-os/mieweb.sample.jsonc` and + `packages/test-app/mieweb.jsonc` comments showing the reserved shape (commented out). + +## Milestone 2 — skopeo image plumbing (cheap-ish, no runtime) + +Image *distribution* only — nothing runs yet. + +- [x] New CLI module `packages/cli/src/images.mjs`: + - [x] `detectBuilder()` — prefer `buildah bud`, fall back to `docker build`; + error clearly if neither is installed. + - [x] `buildImage({ dockerfile, context, tag })` — build to local + `containers-storage:` (buildah) or the Docker daemon. + - [x] `pushImage({ tag, registry })` — `skopeo copy` from local storage to + `docker:////:`; support + `--authfile`/creds from the target's `registry` config; never log secrets. + - [x] `inspectImage(ref)` — `skopeo inspect` for digest pinning (record the + digest so deploys are reproducible). +- [x] `mieweb images push --target mieweb` subcommand wiring in + `packages/cli/src/index.mjs` (build + skopeo copy to Harbor). +- [x] For `--target cloudflare`, delegate to `wrangler containers push` (or + `wrangler deploy`, which builds+pushes) — do **not** reimplement CF's + managed-registry auth with skopeo initially; leave a TODO with the + `wrangler containers images` escape hatch. +- [x] Digest-pin file (e.g. `.mieweb/images.lock.json`): image name → digest per + target, written on push, read on deploy. +- [x] Docs: short "Images" section in `packages/cli` README (or root README) + covering skopeo/buildah prerequisites (`brew install skopeo buildah` / + distro packages). + +## Milestone 3 — Harbor/Forgejo integration points (config only here) + +The cluster work lives in **opensource-server**; this repo only consumes it. + +- [x] Define the Harbor conventions the CLI assumes: project = app name, + repo = container class name (lowercased), tag = git short SHA, plus a + `latest` moving tag. Document in the plan/README. +- [x] Support robot-account auth (`username: 'robot$…'`) and `authFile` in the + registry config; verify skopeo works against Harbor's token service. + *(config + flags done; live-Harbor verification blocked on the cluster — see last box)* +- [x] Forgejo Actions workflow sketch (checked into the app repo, not executed + here): build with buildah, `skopeo copy` to Harbor, run conformance. + Add as a commented example under `packages/test-app/` or docs. + → `packages/test-app/forgejo-images.example.yml` +- [ ] Coordinate with opensource-server: record the Harbor URL + CA expectations + once the standalone cluster deploy lands (blocker for end-to-end testing; + until then use a local Harbor via docker-compose or `skopeo copy` to + `oci-archive:` in tests). + +## Milestone 4 — Local/mieweb container runtime adapter (deferred, the expensive part) + +Do **not** start until an app actually needs a container workload. + +- [ ] `packages/cloud-local/src/adapters/container-docker.mjs`: implement + `ctx.container` (start/stop/ports/monitor/signal) for the in-proc DO + registry by driving a local Docker/Podman daemon; `sleepAfter` → idle + timer → stop; `onStart`/`onStop`/`onError` hooks. +- [ ] Vendor or depend on `@cloudflare/containers` so the app's + `class X extends Container` works unchanged off-Cloudflare. +- [ ] `mieweb` target: same adapter pointed at the cluster's container host + (details TBD with opensource-server — possibly Podman over SSH or a k8s + shim; decide then). +- [ ] Pull images with skopeo from Harbor into local storage before start. +- [ ] Conformance: add a container section to the test-app worker + harness + (`packages/test-app/worker/index.mjs`, `harness/run.mjs`) exercising + start, HTTP round-trip, sleepAfter, and stop across targets. +- [ ] Document divergences: ephemeral disk, SIGTERM→SIGKILL (15 min on CF), + cold-start behavior, placement — local emulation is single-node + best-effort, like the DO adapter. + +--- + +## Documentation & proposed user-facing commands + +What we ship as docs (root `README.md` "Containers" section + `packages/cli` +README "Images" section + sample-config comments), and the exact UX they show. + +### App-side code (unchanged from Cloudflare) + +The docs lead with the point of the whole repo: the app writes *stock +Cloudflare* container code and it stays portable. + +```js +// worker/index.mjs — same on every target +import { Container, getContainer } from '@cloudflare/containers'; + +export class JobRunner extends Container { + defaultPort = 4000; + sleepAfter = '10m'; +} + +export default { + async fetch(request, env) { + return getContainer(env.JOB_RUNNER, 'session-42').fetch(request); + }, +}; +``` + +```jsonc +// wrangler.jsonc — stays the single source of truth for bindings +{ + "containers": [{ "class_name": "JobRunner", "image": "./Dockerfile", "max_instances": 5 }], + "durable_objects": { "bindings": [{ "class_name": "JobRunner", "name": "JOB_RUNNER" }] }, + "migrations": [{ "tag": "v1", "new_sqlite_classes": ["JobRunner"] }] +} +``` + +```jsonc +// mieweb.jsonc — sidecar: only registry + driver hints, per target +{ + "targets": { + "mieweb": { + "registry": { + "url": "harbor.os.mieweb.org", + "project": "cloud-apps", + "username": "robot$cloud-apps+ci", + "authFile": "~/.config/mieweb/harbor-auth.json" + }, + "bindings": { "JOB_RUNNER": { "driver": "docker" } } // M4; "unsupported" until then + } + } +} +``` + +### Proposed commands + +Image lifecycle (Milestone 2): + +```sh +# Build the image(s) declared in wrangler.jsonc's `containers` array. +# Uses buildah if present, else docker. Tags with the git short SHA. +mieweb images build + +# Build + skopeo-copy to the active target's registry, pin digest in +# .mieweb/images.lock.json. On --target cloudflare this delegates to +# `wrangler containers push` instead of skopeo. +mieweb images push --target mieweb # → docker://harbor.os.mieweb.org/cloud-apps/jobrunner: +mieweb images push --target cloudflare # → wrangler containers push + +# Inspect what a target would run (skopeo inspect; shows digest, layers, arch). +mieweb images inspect --target mieweb JOB_RUNNER + +# Show the digest lockfile vs. what's live in the registry. +mieweb images status +``` + +Registry auth (Milestone 3 — thin wrappers over skopeo's auth, never store +secrets in mieweb.jsonc committed files): + +```sh +mieweb registry login --target mieweb # → skopeo login harbor.os.mieweb.org (writes authFile) +mieweb registry logout --target mieweb +``` + +Dev & deploy (Milestones 1/4 — no new verbs, existing ones grow container +awareness): + +```sh +mieweb --target cloudflare dev # wrangler dev: builds + runs the image locally (needs Docker) +mieweb --target cloudflare deploy # wrangler deploy: builds, pushes, rolls out — unchanged + +mieweb --target local dev # M1: boots; container binding throws UnsupportedBindingError on use + # M4: skopeo-pulls image, docker-runs it, proxies getContainer().fetch() + +mieweb --target mieweb deploy # M4: push to Harbor + start on the cluster's container host +``` + +Debugging (documented, not built — these are the underlying tools): + +```sh +wrangler containers list # live CF instances +wrangler containers ssh # shell into a running CF container instance +docker exec -it mieweb-jobrunner-session-42 sh # local-target instance (M4 naming convention) +skopeo inspect docker://harbor.os.mieweb.org/cloud-apps/jobrunner:latest +``` + +### Documentation checklist + +- [x] Root `README.md`: add Containers row to the binding table + a short + "Containers" section with the app-side example above and the + target-support matrix (cloudflare ✅ / local ⏳ M4 / mieweb ⏳ M4). +- [x] `packages/cli` README (create if absent): "Images" section — prerequisites + (`brew install skopeo buildah`), the `mieweb images …` and + `mieweb registry …` commands, lockfile semantics, Harbor conventions + (project/repo/tag naming from Milestone 3). +- [x] `packages/cloud-types/src/index.ts` header table row + doc comments + (part of M0, listed here for completeness). +- [x] Sample configs (`packages/cloud-os/mieweb.sample.jsonc`, + `packages/test-app/mieweb.jsonc`): commented-out registry + container + binding blocks matching the example above. +- [ ] "Debugging containers" subsection: the `wrangler containers ssh` / + `docker exec` / `skopeo inspect` table above, per target. +- [ ] Divergence notes (with M4): ephemeral disk, SIGTERM→SIGKILL window, + cold starts, single-node local emulation. + +--- + +## Sequencing / acceptance + +| Milestone | Depends on | Done when | +| --------- | ---------- | --------- | +| 0 | — | types compile; READMEs updated; test-app unaffected | +| 1 | 0 | schema validates a container-reserved config; `local` dev throws `UnsupportedBindingError` on use, not at boot | +| 2 | 1 | `mieweb images push` builds and skopeo-copies to a registry; digest lockfile written | +| 3 | 2 + opensource-server Harbor | push to real Harbor with robot account succeeds from CI sketch | +| 4 | 2 + a real consumer app | conformance container section green on `local` (and `mieweb` when cluster host exists) | diff --git a/packages/README.md b/packages/README.md index ec7c4cc..d87fe2b 100644 --- a/packages/README.md +++ b/packages/README.md @@ -16,7 +16,7 @@ implement the same Cloudflare-shaped contract. | Package | Role | | ------- | ---- | -| [`cloud-types`](cloud-types) | The portable contracts (`CloudDatabase`, `CloudBucket`, `CloudKV`, `CloudQueue`, `CloudStatefulNamespace`, `CloudVectorIndex`, `CloudAI`) plus `UnsupportedBindingError`. On Cloudflare these are exact aliases of the native binding types, so `worker/env.ts` keeps compiling unchanged. | +| [`cloud-types`](cloud-types) | The portable contracts (`CloudDatabase`, `CloudBucket`, `CloudKV`, `CloudQueue`, `CloudStatefulNamespace`, `CloudVectorIndex`, `CloudAI`, `CloudContainerNamespace`) plus `UnsupportedBindingError`. On Cloudflare these are exact aliases of the native binding types, so `worker/env.ts` keeps compiling unchanged. | | [`cloud-workers`](cloud-workers) | The `DurableObject` base. Backs the **`mieweb:workers`** virtual import: re-exports `cloudflare:workers` on Cloudflare (workerd export condition), pure-JS base everywhere else. | | [`cloud`](cloud) | Umbrella entry. Re-exports the contracts + `DurableObject` from one stable import surface (`@mieweb/cloud`). | | [`cloud-local`](cloud-local) | Local/Node **adapters** (the POC): D1→SQLite, R2→filesystem, KV→in-memory, Queues→in-process, Durable Objects→in-process registry. Vectorize/Workers AI surface explicit `UnsupportedBindingError`. Includes the Node **host harness** that runs the unchanged worker handler and a migration runner. | diff --git a/packages/cli/README.md b/packages/cli/README.md new file mode 100644 index 0000000..27b438b --- /dev/null +++ b/packages/cli/README.md @@ -0,0 +1,68 @@ +# `@mieweb/cli` — the `mieweb` command + +Target-aware wrapper over `wrangler`. On the `cloudflare` target (default) +every command is forwarded verbatim to `wrangler`; on `local`/`mieweb` the CLI +runs your unchanged worker on the Node host harness backed by the matching +adapters. See the [root README](../../README.md) for the full model. + +```sh +mieweb [--target ] [...args] +``` + +## Images (Cloudflare Containers) + +Build container images once, distribute them with **skopeo** +(see [container-plan.md](../../container-plan.md)). + +**Prerequisites:** `skopeo` plus a builder — `buildah` (preferred) or `docker`. + +```sh +brew install skopeo buildah # macOS +# apt/dnf install skopeo buildah # Linux +``` + +The CLI reads the `containers` array in `wrangler.jsonc` (class → Dockerfile) +and the per-target `registry` block in `mieweb.jsonc`: + +```jsonc +// mieweb.jsonc +{ + "targets": { + "mieweb": { + "registry": { + "url": "harbor.os.mieweb.org", + "project": "cloud-apps", + "username": "robot$cloud-apps+ci", + "authFile": "~/.config/mieweb/harbor-auth.json" + } + } + } +} +``` + +### Commands + +```sh +mieweb images build # build every containers[] image (buildah/docker) +mieweb images push --target mieweb # build + skopeo copy → registry, pin digests +mieweb images push # cloudflare target: delegates to `wrangler containers push` +mieweb images inspect CONVERTER # skopeo inspect by binding name or class name +mieweb images status # lockfile pins vs. what's live in the registry + +mieweb registry login --target mieweb # skopeo login (writes authFile; password prompted, never argv) +mieweb registry logout --target mieweb +``` + +### Conventions + +- **Naming:** `docker:////:`, + plus a `latest` moving tag. `project` defaults to the wrangler app `name`. +- **Lockfile:** pushes pin the manifest digest per class per target in + `.mieweb/images.lock.json` (commit it — it's what makes deploys reproducible). + `mieweb images status` reports drift between the pin and the registry's `latest`. +- **Auth:** prefer `authFile` (written by `mieweb registry login`) over inline + credentials; secrets are never passed on the command line or logged. + Harbor robot accounts (`robot$project+name`) are the expected CI identity. +- **Cloudflare:** the managed registry is wrangler's job — `images push` on the + `cloudflare` target hands off to `wrangler containers push` rather than + reimplementing its auth with skopeo. diff --git a/packages/cli/mieweb-config.schema.json b/packages/cli/mieweb-config.schema.json index a106c8b..50f4fed 100644 --- a/packages/cli/mieweb-config.schema.json +++ b/packages/cli/mieweb-config.schema.json @@ -22,6 +22,19 @@ "properties": { "runtime": { "type": "string" }, "port": { "type": "number" }, + "registry": { + "type": "object", + "description": "OCI registry the CLI pushes container images to for this target (skopeo copy destination). Harbor for the mieweb target; unused on cloudflare (wrangler's managed registry).", + "properties": { + "url": { "type": "string", "description": "Registry host, e.g. harbor.os.mieweb.org." }, + "project": { "type": "string", "description": "Registry project/namespace images are pushed under." }, + "username": { "type": "string", "description": "Registry user (e.g. a Harbor robot account: robot$project+name)." }, + "authFile": { "type": "string", "description": "Path to a containers-auth.json written by `skopeo login` / `mieweb registry login`. Preferred over inline credentials." }, + "insecureSkipTlsVerify": { "type": "boolean", "description": "Disable TLS verification (dev registries only)." } + }, + "required": ["url"], + "additionalProperties": true + }, "bindings": { "type": "object", "additionalProperties": { @@ -42,6 +55,7 @@ "s3", "valkey", "valkey-queue", + "docker", "unsupported" ] }, diff --git a/packages/cli/src/config.mjs b/packages/cli/src/config.mjs index a16d321..311c511 100644 --- a/packages/cli/src/config.mjs +++ b/packages/cli/src/config.mjs @@ -13,6 +13,8 @@ import { parseJsonc } from './jsonc.mjs'; * @property {Record} wrangler parsed wrangler.jsonc * @property {Record} raw parsed mieweb.jsonc (or {} ) * @property {Record} targetConfig adapter config for the active target + * @property {Array>} containers wrangler `containers` entries (Cloudflare Containers) + * @property {Record>} containerBindings DO binding name → its wrangler `containers` entry, for container-backed bindings */ /** @@ -52,6 +54,26 @@ export function loadConfig(opts = {}) { const targetConfig = (raw.targets && raw.targets[target]) || {}; + // Cloudflare Containers: wrangler's `containers` array pairs a DO class with + // an image; a DO binding whose class appears there is a container binding. + // (Reserved surface — see container-plan.md. wrangler handles these natively + // on the cloudflare target; other targets throw UnsupportedBindingError on + // use until a runtime adapter exists.) + const containers = Array.isArray(wrangler.containers) + ? /** @type {Array>} */ (wrangler.containers) + : []; + /** @type {Record>} */ + const containerBindings = {}; + const doBindings = + /** @type {{ bindings?: Array<{ name: string, class_name: string }> }} */ ( + wrangler.durable_objects + )?.bindings ?? []; + for (const c of containers) { + for (const b of doBindings) { + if (b.class_name === c.class_name) containerBindings[b.name] = c; + } + } + return { configPath: miewebPath ?? wranglerPath, root, @@ -60,6 +82,8 @@ export function loadConfig(opts = {}) { wrangler, raw, targetConfig, + containers, + containerBindings, }; } diff --git a/packages/cli/src/images.mjs b/packages/cli/src/images.mjs new file mode 100644 index 0000000..9e241fa --- /dev/null +++ b/packages/cli/src/images.mjs @@ -0,0 +1,331 @@ +import { spawn } from 'node:child_process'; +import { readFileSync, writeFileSync, mkdirSync, existsSync } from 'node:fs'; +import { resolve, dirname, join } from 'node:path'; +import { homedir } from 'node:os'; + +/** + * `mieweb images …` / `mieweb registry …` — container image plumbing. + * + * Build once, distribute with skopeo (container-plan.md Milestone 2): + * + * mieweb images build build every wrangler `containers` image + * mieweb images push [--target mieweb] build + skopeo copy to the target registry + * mieweb images inspect skopeo inspect what the target would run + * mieweb images status digest lockfile vs. registry + * mieweb registry login|logout skopeo login against the target registry + * + * On `--target cloudflare`, push delegates to `wrangler containers push` + * (Cloudflare's managed registry has its own auth dance) — skopeo is only used + * for self-managed registries (Harbor on the `mieweb` target, local dev). + * + * Naming convention (container-plan.md Milestone 3): + * docker:////: + * plus a `latest` moving tag. Pushed digests are pinned in + * `.mieweb/images.lock.json` so deploys are reproducible. + */ + +/* ---------------------------------------------------------------- helpers */ + +/** + * Run a command, inheriting stdio (interactive-friendly). + * @param {string} cmd @param {string[]} args @param {{cwd?: string}} [opts] + * @returns {Promise} + */ +function run(cmd, args, opts = {}) { + return new Promise((res, rej) => { + const child = spawn(cmd, args, { cwd: opts.cwd, stdio: 'inherit' }); + child.on('error', rej); + child.on('exit', (code) => res(code ?? 0)); + }); +} + +/** + * Run a command capturing stdout (for skopeo inspect etc.). + * @param {string} cmd @param {string[]} args @param {{cwd?: string}} [opts] + * @returns {Promise<{code: number, stdout: string, stderr: string}>} + */ +function capture(cmd, args, opts = {}) { + return new Promise((res, rej) => { + const child = spawn(cmd, args, { cwd: opts.cwd, stdio: ['ignore', 'pipe', 'pipe'] }); + let stdout = ''; + let stderr = ''; + child.stdout.on('data', (d) => { stdout += d; }); + child.stderr.on('data', (d) => { stderr += d; }); + child.on('error', rej); + child.on('exit', (code) => res({ code: code ?? 0, stdout, stderr })); + }); +} + +/** @param {string} cmd */ +async function commandExists(cmd) { + const { code } = await capture('sh', ['-c', `command -v ${cmd}`]); + return code === 0; +} + +/** + * Prefer buildah, fall back to docker. Determines both the build command and + * the skopeo *source* transport for the built image. + * @returns {Promise<{ name: 'buildah'|'docker', srcTransport: (ref: string) => string }>} + */ +export async function detectBuilder() { + if (await commandExists('buildah')) { + return { name: 'buildah', srcTransport: (ref) => `containers-storage:${ref}` }; + } + if (await commandExists('docker')) { + return { name: 'docker', srcTransport: (ref) => `docker-daemon:${ref}` }; + } + throw new Error( + 'mieweb images: neither buildah nor docker found on PATH. ' + + 'Install one of them (and skopeo) — e.g. `brew install buildah skopeo`.', + ); +} + +/** Git short SHA of HEAD (falls back to "dev" outside a repo). */ +async function gitShortSha(cwd) { + const { code, stdout } = await capture('git', ['rev-parse', '--short', 'HEAD'], { cwd }); + return code === 0 ? stdout.trim() : 'dev'; +} + +/** Expand a leading `~` (authFile paths). @param {string} p */ +function expandHome(p) { + return p.startsWith('~') ? join(homedir(), p.slice(1)) : p; +} + +/** + * The wrangler `containers` entries that are locally built (image is a + * Dockerfile path, not a remote ref). + * @param {import('./config.mjs').MiewebConfig} config + */ +function buildableContainers(config) { + return (config.containers ?? []).filter( + (c) => typeof c.image === 'string' && !c.image.includes('://') && !c.image.startsWith('registry.'), + ); +} + +/** + * Registry config for the active target (mieweb.jsonc `targets..registry`). + * @param {import('./config.mjs').MiewebConfig} config + */ +function registryFor(config) { + const reg = config.targetConfig?.registry; + if (!reg || typeof reg.url !== 'string') { + throw new Error( + `mieweb images: no registry configured for target "${config.target}". ` + + `Add targets.${config.target}.registry = { url, project, … } to mieweb.jsonc.`, + ); + } + return reg; +} + +/** + * Fully-qualified repo ref (no tag) for a container class on a registry. + * @param {{ url: string, project?: string }} reg + * @param {Record} c wrangler containers entry + * @param {import('./config.mjs').MiewebConfig} config + */ +function repoRef(reg, c, config) { + const project = reg.project ?? config.wrangler?.name ?? 'mieweb'; + return `${reg.url}/${project}/${String(c.class_name).toLowerCase()}`; +} + +/** skopeo auth/TLS flags for a destination registry. @param {any} reg @param {'src'|'dest'} side */ +function skopeoAuthFlags(reg, side) { + const flags = []; + if (reg.authFile) flags.push(`--${side}-authfile`, expandHome(reg.authFile)); + if (reg.insecureSkipTlsVerify) flags.push(`--${side}-tls-verify=false`); + return flags; +} + +/* ------------------------------------------------------------- lockfile */ + +/** @param {string} root */ +function lockPath(root) { + return resolve(root, '.mieweb/images.lock.json'); +} + +/** @param {string} root */ +function readLock(root) { + const p = lockPath(root); + if (!existsSync(p)) return {}; + try { + return JSON.parse(readFileSync(p, 'utf8')); + } catch { + return {}; + } +} + +/** @param {string} root @param {Record} lock */ +function writeLock(root, lock) { + const p = lockPath(root); + mkdirSync(dirname(p), { recursive: true }); + writeFileSync(p, `${JSON.stringify(lock, null, 2)}\n`); +} + +/* ------------------------------------------------------------ operations */ + +/** + * Build one container image locally. + * @param {{ name: 'buildah'|'docker' }} builder + * @param {Record} c wrangler containers entry + * @param {string} root + * @param {string} tag local ref, e.g. mieweb/jobrunner:abc1234 + */ +export async function buildImage(builder, c, root, tag) { + const dockerfile = resolve(root, c.image); + const context = dirname(dockerfile); + const args = + builder.name === 'buildah' + ? ['bud', '-f', dockerfile, '-t', tag, context] + : ['build', '-f', dockerfile, '-t', tag, context]; + const code = await run(builder.name, args, { cwd: root }); + if (code !== 0) throw new Error(`mieweb images: ${builder.name} build failed for ${c.class_name} (exit ${code})`); +} + +/** + * skopeo copy a locally built image to the registry (sha tag + latest). + * @returns {Promise} the pushed manifest digest + */ +export async function pushImage(builder, reg, localRef, remoteRepo, sha) { + if (!(await commandExists('skopeo'))) { + throw new Error('mieweb images: skopeo not found on PATH (`brew install skopeo`).'); + } + for (const tag of [sha, 'latest']) { + const dest = `docker://${remoteRepo}:${tag}`; + const code = await run('skopeo', [ + 'copy', + ...skopeoAuthFlags(reg, 'dest'), + builder.srcTransport(localRef), + dest, + ]); + if (code !== 0) throw new Error(`mieweb images: skopeo copy to ${dest} failed (exit ${code})`); + } + return inspectDigest(reg, `${remoteRepo}:${sha}`); +} + +/** skopeo inspect a remote ref and return its digest. */ +async function inspectDigest(reg, ref) { + const { code, stdout, stderr } = await capture('skopeo', [ + 'inspect', + ...skopeoAuthFlags(reg, 'src'), + `docker://${ref}`, + ]); + if (code !== 0) throw new Error(`mieweb images: skopeo inspect docker://${ref} failed: ${stderr.trim()}`); + return JSON.parse(stdout).Digest; +} + +/* -------------------------------------------------------------- commands */ + +/** + * Entry point for `mieweb images [...]`. + * @param {string[]} args after "images" + * @param {import('./config.mjs').MiewebConfig} config + * @returns {Promise} + */ +export async function runImagesCommand(args, config) { + const sub = args[0]; + const containers = buildableContainers(config); + if (containers.length === 0) { + console.error('mieweb images: no `containers` entries with a local Dockerfile in wrangler.jsonc.'); + return 1; + } + + const sha = await gitShortSha(config.root); + + if (sub === 'build' || sub === 'push') { + const builder = await detectBuilder(); + /** @type {Record} */ + const lock = readLock(config.root); + + for (const c of containers) { + const localRef = `mieweb/${String(c.class_name).toLowerCase()}:${sha}`; + console.log(`[mieweb] building ${c.class_name} → ${localRef} (${builder.name})`); + await buildImage(builder, c, config.root, localRef); + + if (sub === 'push') { + const reg = registryFor(config); + const repo = repoRef(reg, c, config); + console.log(`[mieweb] pushing ${localRef} → ${repo}:{${sha},latest} (skopeo)`); + const digest = await pushImage(builder, reg, localRef, repo, sha); + lock[c.class_name] = { ...(lock[c.class_name] ?? {}), [config.target]: { repo, tag: sha, digest } }; + console.log(`[mieweb] pinned ${c.class_name}@${digest}`); + } + } + if (sub === 'push') writeLock(config.root, lock); + return 0; + } + + if (sub === 'inspect') { + const reg = registryFor(config); + const nameArg = args[1]; + const targets = nameArg + ? containers.filter( + (c) => c.class_name === nameArg || config.containerBindings?.[nameArg]?.class_name === c.class_name, + ) + : containers; + if (targets.length === 0) { + console.error(`mieweb images: no container matches "${nameArg}".`); + return 1; + } + for (const c of targets) { + const code = await run('skopeo', [ + 'inspect', + ...skopeoAuthFlags(reg, 'src'), + `docker://${repoRef(reg, c, config)}:latest`, + ]); + if (code !== 0) return code; + } + return 0; + } + + if (sub === 'status') { + const lock = readLock(config.root); + if (Object.keys(lock).length === 0) { + console.log('mieweb images: no lockfile yet (.mieweb/images.lock.json) — run `mieweb images push`.'); + return 0; + } + const reg = registryFor(config); + for (const [className, perTarget] of Object.entries(lock)) { + const pinned = perTarget?.[config.target]; + if (!pinned) { + console.log(`${className}: no pin for target "${config.target}"`); + continue; + } + try { + const live = await inspectDigest(reg, `${pinned.repo}:latest`); + const match = live === pinned.digest ? 'in sync' : `DRIFT (latest=${live})`; + console.log(`${className}: pinned ${pinned.tag} ${pinned.digest} — ${match}`); + } catch (err) { + console.log(`${className}: pinned ${pinned.tag} ${pinned.digest} — inspect failed: ${/** @type {Error} */ (err).message}`); + } + } + return 0; + } + + console.error('Usage: mieweb images '); + return 1; +} + +/** + * Entry point for `mieweb registry `. + * @param {string[]} args after "registry" + * @param {import('./config.mjs').MiewebConfig} config + * @returns {Promise} + */ +export async function runRegistryCommand(args, config) { + const sub = args[0]; + if (sub !== 'login' && sub !== 'logout') { + console.error('Usage: mieweb registry '); + return 1; + } + if (!(await commandExists('skopeo'))) { + console.error('mieweb registry: skopeo not found on PATH (`brew install skopeo`).'); + return 1; + } + const reg = registryFor(config); + const flags = []; + if (reg.authFile) flags.push('--authfile', expandHome(reg.authFile)); + if (reg.insecureSkipTlsVerify) flags.push('--tls-verify=false'); + if (sub === 'login' && reg.username) flags.push('--username', reg.username); + // Interactive: skopeo prompts for the password itself — never passed via argv. + return run('skopeo', [sub, ...flags, reg.url]); +} diff --git a/packages/cli/src/index.mjs b/packages/cli/src/index.mjs index 79a4b8f..fc9968f 100755 --- a/packages/cli/src/index.mjs +++ b/packages/cli/src/index.mjs @@ -22,6 +22,7 @@ import { loadConfig } from './config.mjs'; import { delegateToWrangler } from './cloudflare.mjs'; import { runHostTarget } from './local.mjs'; import { runInit } from './init.mjs'; +import { runImagesCommand, runRegistryCommand } from './images.mjs'; /** Read this CLI's version from its package.json. */ function miewebVersion() { @@ -74,6 +75,25 @@ async function main(argv) { const config = loadConfig({ overrideTarget }); + // Container image plumbing (build once, skopeo copy — container-plan.md M2). + // `images push` on the cloudflare target delegates to wrangler's managed + // registry; every other target goes through skopeo to its own registry. + if (args[0] === 'images') { + if (config.target === 'cloudflare' && args[1] === 'push') { + return delegateToWrangler(['containers', 'push', ...args.slice(2)], { cwd: config.root }); + } + return runImagesCommand(args.slice(1), config).catch((err) => { + console.error(err?.message ?? err); + return 1; + }); + } + if (args[0] === 'registry') { + return runRegistryCommand(args.slice(1), config).catch((err) => { + console.error(err?.message ?? err); + return 1; + }); + } + if (config.target === 'cloudflare') { // Reference path: hand everything to wrangler untouched. return delegateToWrangler(args, { cwd: config.root }); @@ -112,6 +132,8 @@ function printHelp() { ' mieweb deploy Deploy (cloudflare only).', ' mieweb tail Stream logs (cloudflare only).', ' mieweb d1 migrations apply Apply ./migrations to the target DB.', + ' mieweb images build|push|inspect|status Build & skopeo-push container images.', + ' mieweb registry login|logout skopeo login to the target registry.', '', 'Selecting a target:', ' mieweb --target local dev Flag form.', diff --git a/packages/cloud-local/src/adapters/unsupported.mjs b/packages/cloud-local/src/adapters/unsupported.mjs index 1f43962..7ae443a 100644 --- a/packages/cloud-local/src/adapters/unsupported.mjs +++ b/packages/cloud-local/src/adapters/unsupported.mjs @@ -7,12 +7,15 @@ * * @param {string} bindingName e.g. 'SEARCH_INDEX' * @param {string} target e.g. 'local' + * @param {string} [hint] optional guidance appended to the error message * @returns {any} a proxy that throws on use */ -export function createUnsupportedBinding(bindingName, target) { +export function createUnsupportedBinding(bindingName, target, hint) { const fail = () => { const err = new Error( - `Binding "${bindingName}" is not supported on target "${target}".`, + `Binding "${bindingName}" is not supported on target "${target}"${ + hint ? `: ${hint}` : '.' + }`, ); err.name = 'UnsupportedBindingError'; throw err; diff --git a/packages/cloud-local/src/host.mjs b/packages/cloud-local/src/host.mjs index 77577ac..da5e513 100644 --- a/packages/cloud-local/src/host.mjs +++ b/packages/cloud-local/src/host.mjs @@ -1,6 +1,7 @@ import { pathToFileURL } from 'node:url'; import { resolve } from 'node:path'; import { createCloudEnv, settleEnv } from './index.mjs'; +import { createUnsupportedBinding } from './adapters/unsupported.mjs'; /** * Node host harness: runs the **unchanged** worker handler off Cloudflare. @@ -66,6 +67,24 @@ export async function startLocalHost({ config }) { }); await settleEnv(env); + // Container-backed DO bindings (Cloudflare Containers) have no off-Cloudflare + // runtime yet: boot cleanly, throw UnsupportedBindingError on first use. + // (Reserved surface — see container-plan.md Milestone 4.) + const containerClasses = new Set( + (Array.isArray(wrangler.containers) ? wrangler.containers : []).map( + (/** @type {{ class_name?: string }} */ c) => c.class_name, + ), + ); + for (const b of doBindings) { + if (containerClasses.has(b.class_name) && env[b.name] === undefined) { + env[b.name] = createUnsupportedBinding( + b.name, + config.target ?? 'local', + 'container runtime adapter not yet implemented (see container-plan.md)', + ); + } + } + // 3. Serve fetch over HTTP. const port = targetConfig.port ?? wrangler?.dev?.port ?? 8787; const { serve } = await import('@hono/node-server'); diff --git a/packages/cloud-os/mieweb.sample.jsonc b/packages/cloud-os/mieweb.sample.jsonc index 57ce41b..e1ff9a4 100644 --- a/packages/cloud-os/mieweb.sample.jsonc +++ b/packages/cloud-os/mieweb.sample.jsonc @@ -13,6 +13,14 @@ "mieweb": { "runtime": "node", "port": 8787, + // Cloudflare Containers (reserved — see container-plan.md): where + // `mieweb images push --target mieweb` skopeo-copies images to. + // "registry": { + // "url": "harbor.os.mieweb.org", + // "project": "cloud-apps", + // "username": "robot$cloud-apps+ci", + // "authFile": "~/.config/mieweb/harbor-auth.json" + // }, "bindings": { // D1 → libSQL/sqld (SQLite-compatible: keeps FTS5). Migrations apply // unchanged. Add `authToken` for a secured server. diff --git a/packages/cloud-types/src/index.ts b/packages/cloud-types/src/index.ts index 0b9f654..eb86aa5 100644 --- a/packages/cloud-types/src/index.ts +++ b/packages/cloud-types/src/index.ts @@ -22,6 +22,7 @@ * | Durable Objects | Artipods / stateful container | `CloudStatefulNamespace` | * | Vectorize | Footnote vector index | `CloudVectorIndex` | * | Workers AI | Ozwell AI gateway | `CloudAI` | + * | Containers | DO-controlled Linux container | `CloudContainerNamespace` | */ /** SQLite-compatible database (Cloudflare D1). */ @@ -57,6 +58,21 @@ export type CloudVectorIndex = VectorizeIndex; /** AI model gateway ("Ozwell" / Cloudflare Workers AI). */ export type CloudAI = Ai; +/** + * Namespace of Durable-Object-controlled Linux containers (Cloudflare + * Containers). A container binding *is* a DO namespace: the app-side class + * (`class MyContainer extends Container`, from `@cloudflare/containers`) + * extends a Durable Object, and `wrangler.jsonc` pairs a `containers` array + * entry with an ordinary `durable_objects.bindings` entry. + * + * Cloudflare-only for now: non-Cloudflare adapters surface + * `UnsupportedBindingError` until a portable runtime emulates `ctx.container` + * (see container-plan.md, Milestone 4). + */ +export type CloudContainerNamespace = DurableObjectNamespace; +/** What `getContainer(ns, id)` / `ns.get(id)` returns — a routable stub. */ +export type CloudContainerStub = DurableObjectStub; + /** Per-request lifecycle hook (`waitUntil`, `passThroughOnException`). */ export type CloudExecutionContext = ExecutionContext; diff --git a/packages/test-app/forgejo-images.example.yml b/packages/test-app/forgejo-images.example.yml new file mode 100644 index 0000000..6d15bc0 --- /dev/null +++ b/packages/test-app/forgejo-images.example.yml @@ -0,0 +1,47 @@ +# Forgejo Actions workflow *sketch* for an app consuming @mieweb/cloud with +# containers (container-plan.md Milestone 3). Not executed in this repo — +# copy into a consuming app as .forgejo/workflows/images.yml once the +# opensource-server cluster (Forgejo + Harbor) is up. +# +# Identity: a Harbor robot account (robot$+ci) stored as the +# HARBOR_ROBOT_* secrets. Images are built with buildah (rootless, no daemon) +# and moved with skopeo — same pipeline as `mieweb images push` locally. + +name: images +on: + push: + branches: [main] + +jobs: + build-and-push: + runs-on: docker # a runner image with: node+pnpm, buildah, skopeo, git + steps: + - uses: actions/checkout@v4 + + - name: Login to Harbor (writes the authfile skopeo/mieweb use) + run: | + skopeo login \ + --username "${{ secrets.HARBOR_ROBOT_USER }}" \ + --password-stdin \ + --authfile "$RUNNER_TEMP/harbor-auth.json" \ + harbor.os.mieweb.org <<< "${{ secrets.HARBOR_ROBOT_TOKEN }}" + + - name: Build + push (skopeo copy, digest-pinned) + env: + # mieweb.jsonc's registry.authFile can point at this via env override, + # or set targets.mieweb.registry.authFile to the same path. + MIEWEB_TARGET: mieweb + run: | + pnpm install --frozen-lockfile + pnpm exec mieweb images push --target mieweb + + - name: Conformance against the pushed image (once M4 lands) + run: pnpm exec mieweb --target mieweb test || true # placeholder + + - name: Commit the digest lockfile bump + run: | + git config user.name forgejo-actions + git config user.email actions@os.mieweb.org + git add .mieweb/images.lock.json + git diff --cached --quiet || git commit -m "ci: pin container image digests" + git push diff --git a/packages/test-app/mieweb.jsonc b/packages/test-app/mieweb.jsonc index 5c31fbb..0c9dbeb 100644 --- a/packages/test-app/mieweb.jsonc +++ b/packages/test-app/mieweb.jsonc @@ -20,6 +20,10 @@ // No local model server in CI → surface UnsupportedBindingError, which // the worker reports as a skipped surface. "AI": { "driver": "unsupported" } + // Cloudflare Containers (reserved — container-plan.md M4): a container- + // backed DO binding needs no entry here; the host auto-wires it to + // UnsupportedBindingError until the docker driver lands. Then: + // "CONVERTER": { "driver": "docker" } } },