Add managed worlds control plane: telemetry, import, scenarios, captures

Cargo.toml

@@ -16,6 +16,7 @@ lemmalog-syntax = { path = "crates/syntax", version = "0.1.0" }
 serde = { version = "1", features = ["derive"] }
 serde_json = { version = "1", features = ["float_roundtrip"] }
 sha2 = "0.10"
+regex = "1"
 
 iceberg = { git = "https://github.com/apache/iceberg-rust.git", rev = "4d83bc77dc10cff851a3ef427c451ff977bc3643", optional = true }
 iceberg-catalog-sql = { git = "https://github.com/apache/iceberg-rust.git", rev = "4d83bc77dc10cff851a3ef427c451ff977bc3643", optional = true }

README.md

@@ -49,6 +49,15 @@ Library callers can explicitly save and restore pure-program local checkpoints.
 
 A runnable library example is [`examples/program.rs`](examples/program.rs). Installation invokes native compilation; pure parsing, lowering and registry validation do not. The workspace does not require the MCP feature for library use.
 
+## World control plane
+
+`cargo build --locked --bin ddlog-worlds` builds an explicitly launched owner for
+registered worlds. It provides asynchronous start/stop, persistent instance history,
+real process CPU/RSS, and opt-in native topology through versioned inspection
+contracts. Registration alone never starts execution. See [world lifecycle and
+protocol](docs/worlds.md) and [inspection metadata](docs/inspection.md). Existing
+`ProgramInstance` and shared-host callers remain supported.
+
 ## MCP
 
 ```sh
@@ -92,6 +101,6 @@ cargo build --locked --features mcp --bin lemmalog-ddlog-mcp
 python3 -m unittest discover -s tests -p 'test_*.py' -v
 ```
 
-These tests need Rust and Python 3.11+, but no DDlog compiler or provider. Python host/MCP tests use explicitly simulated graph fixtures. [Native acceptance](docs/building.md#native-acceptance) is a separate operator-configured step. Compatibility fixtures captured before extraction check old registry records, content hashes, generated source, bundled native source and MCP schemas.
+These tests need Rust and Python 3.11+, but no DDlog compiler or provider. `--all-features` includes the Iceberg checkpoint feature, whose pinned `iceberg` revision needs Rust 1.95 (`cargo +1.95.0 …`, see [Iceberg checkpoints](docs/iceberg-checkpoints.md)); the default feature set and `--features mcp` build on the 1.94 toolchain. Python host/MCP tests use explicitly simulated graph fixtures. [Native acceptance](docs/building.md#native-acceptance) is a separate operator-configured step. Compatibility fixtures captured before extraction check old registry records, content hashes, generated source, bundled native source and MCP schemas.
 
 MIT licensed; the upstream copyright notice is retained in [LICENSE](LICENSE).

build.rs

@@ -0,0 +1,31 @@
+//! Records the crate's git identity for the `runtime_info` control-plane verb.
+//! A missing git checkout yields no commit; the runtime reports that truthfully.
+use std::process::Command;
+
+fn git(args: &[&str]) -> Option<String> {
+    let output = Command::new("git").args(args).output().ok()?;
+    if !output.status.success() {
+        return None;
+    }
+    Some(String::from_utf8(output.stdout).ok()?.trim().to_string())
+}
+
+fn main() {
+    println!("cargo:rerun-if-changed=build.rs");
+    for path in [
+        ".git/HEAD",
+        ".git/index",
+        "../.git/modules/ddlog-runtime/HEAD",
+    ] {
+        println!("cargo:rerun-if-changed={path}");
+    }
+    let commit = git(&["rev-parse", "HEAD"]).filter(|hash| hash.len() == 40);
+    let dirty = commit.is_some()
+        && git(&["status", "--porcelain", "--untracked-files=no"])
+            .is_some_and(|status| !status.is_empty());
+    println!(
+        "cargo:rustc-env=DDLOG_RUNTIME_COMMIT={}",
+        commit.unwrap_or_default()
+    );
+    println!("cargo:rustc-env=DDLOG_RUNTIME_DIRTY={dirty}");
+}

docs/building.md

@@ -27,7 +27,7 @@ The smoke test retains the actual MCP requests/responses, generated source, exec
 
 ### Existing operator-managed environments
 
-The driver still accepts `DDLOG_HOME`, `DDLOG_CARGO`, `RUSTC`, `CARGO_HOME`, `CARGO_TARGET_DIR`, `DDLOG_CARGO_CONFIG`, `DDLOG_OFFLINE` and `DDLOG_CARGO_LOCK`. A supplied `DDLOG_CARGO_LOCK` takes precedence over bootstrap's `DDLOG_LOCK_DIR`; otherwise the driver selects the ordinary or Star lock from that directory. The repository root lockfile is for the modern host, not generated programs. Use absolute paths and scope native environment variables to the native server/build process, not the modern host build.
+The driver still accepts `DDLOG_HOME`, `DDLOG_CARGO`, `RUSTC`, `CARGO_HOME`, `CARGO_TARGET_DIR`, `DDLOG_CARGO_CONFIG`, `DDLOG_OFFLINE` and `DDLOG_CARGO_LOCK`. A supplied `DDLOG_CARGO_LOCK` takes precedence over bootstrap's `DDLOG_LOCK_DIR`; otherwise the driver selects the ordinary or Star lock from that directory. A program that imports `lemmalog_star` needs the Star lock (it adds the `types__lemmalog_star` workspace member): the driver refuses a `DDLOG_CARGO_LOCK` that does not name that member before invoking cargo, naming both variables, rather than failing inside `cargo build --locked`. Control planes that run both kinds of program should set `DDLOG_LOCK_DIR` and leave `DDLOG_CARGO_LOCK` unset. The repository root lockfile is for the modern host, not generated programs. Use absolute paths and scope native environment variables to the native server/build process, not the modern host build.
 
 The library bundles `src/star/lemmalog_star.dl` and `.rs` as text and writes them beside generated source when selected. The generated source binds their hashes. This native Rust code compiles in DDlog's generated context, including `Weight` and `ddlog_std`; it is not an independently compiled workspace crate.
 

docs/inspection.md

@@ -0,0 +1,140 @@
+# Inspection metadata version 1
+
+`inspection::InspectionMetadata` is a serializable, validated authored contract.
+Call `validate` before registering a definition and `validate_with_graph` when
+binding it to a live native graph. Unsupported schema versions fail validation;
+a reader must not silently interpret them as version 1.
+
+An authored group has a stable `id` and `member_key` (the composition member's
+qualified identity). Its repository/revision and optional one-based source range
+identify authored provenance. A revision is caller-supplied provenance, not proof
+that a remote repository or release exists. This module does not query Git.
+
+`authoredGroups` and `memberIds` match the observer's existing grouping field names.
+`memberIds` are exact native IDs, including worker distinctions. Groups are separate
+from `NativeGraph.nodes`; they never become operators. Native operators retain
+`id`, `worker`, `address`, `name`, and `debug`. Channels retain `id`, `worker`,
+`scope`, `source`, `target`, `source_port`, and `target_port`. Validation is read-only.
+Native IDs and addresses must never be reconstructed from display labels.
+
+Ports carry a stable authored ID, name, direction, and source-language `data_type`.
+A port can map to several native operator/index pairs. Mapping indexes are native
+port indexes, not layout port IDs. Live validation requires a matching channel in
+the stated direction. Unconnected ports cannot be verified from channel data and
+must have no native mapping until supporting observations exist.
+
+Version 1 groups are flat and disjoint: overlapping native membership is rejected.
+Registered definitions may have no native member IDs or mappings; registration is
+not execution. Live bindings are execution-specific and must be revalidated after
+recompilation or restart. Clients still decide whether a group can be laid out
+within native scopes; visual geometry is deliberately not part of this contract.
+No memory usage, CPU usage, liveness, or process ownership is inferred from a graph.
+
+## Membership by match
+
+Native operator ids are not stable across builds, so a group may declare
+`member_match` instead of (or in addition to) `memberIds`, and a port may declare
+`native_match` instead of `native_ports`. Both are omitted from the wire form when
+absent, so definitions without them keep their content hash.
+
+- `member_match: {scope_name?, debug_pattern?}` — at least one. `scope_name` names a
+  native scope (an operator with nested children, e.g. a `region_named` phase) and
+  claims every matching scope with its whole subtree. `debug_pattern` is a regular
+  expression searched in the operator `debug` text (the DDlog profiler context).
+- `native_match: {debug_pattern, index}` — the port binds to native port `index` of
+  the one group member whose `debug` matches.
+
+`validate()` (registration time) checks shape only for match-based groups: the
+pattern compiles, `scope_name` is non-empty, ids are unique. Membership is resolved
+by `resolve(graph)` when a snapshot is taken, deterministically:
+
+1. Groups are processed in declaration order. Explicit `memberIds` and every
+   `scope_name` match claim first; a claim on an operator already owned by another
+   group is a mapping error.
+2. Then `debug_pattern` matches, again in declaration order, claim only operators
+   nobody claimed yet.
+3. Zero matches for a group, a `native_match` that resolves to zero or several
+   members, a port mapped outside its group, a member whose native children are not
+   all members (a split scope), or top-level members under different native parents
+   are mapping errors naming the group.
+4. The resolved metadata then passes the existing live checks (ids exist, ports are
+   observed on channels in the stated direction).
+
+A snapshot reports the resolved metadata (`authoredGroups[*].memberIds` are the
+resolved ids, `ports[*].native_ports` include the matched binding) and
+`mapping_error: null`, or the authored metadata unchanged with the error text.
+Resolution is a pure function of the metadata and the observed graph; it never
+consults the registry or invents operators.
+
+## Native capture build hook
+
+The standard build driver installs `native/observer.rs` into generated DDlog Rust.
+`scripts/install-observer.py` accepts the generated project directory and verifies
+both pinned generator patch sites before writing. Repeated installation is a no-op;
+changed/duplicate sites fail rather than compiling without observation support.
+Python 3 is required by this build step. Runtime logging and debug regions remain
+opt-in through `DDLOG_OBSERVER_FILE`; building alone does not capture anything.
+
+Capture records are actual Timely, progress, and differential events. The writer
+limits each worker to a byte budget (`DDLOG_OBSERVER_BYTES`, default 64 MiB; values
+below 4096 or non-integers fail the install). Without rotation it emits a
+`capture_status` record with `status: truncated` and `reason: worker_byte_limit`
+when its budget is exhausted and writes nothing further. With
+`DDLOG_OBSERVER_ROTATE=1` it instead truncates the shared file to zero, writes
+`{"stream":"capture_status","status":"rotated","bytes":N}` as the first line of the
+fresh file and continues; if truncation fails it falls back to the truncated marker
+(`reason: rotation_failed`). Consumers must report incomplete capture when a
+truncation marker appears; absence of additional events is not proof of idle
+execution. Rotation discards earlier bytes of every worker (the budget is per
+worker, the file is shared), so a reader must have ingested topology before the
+first rotation; the runtime's reader keeps what it ingested and restarts at offset
+zero when the file shrinks. The runtime must select a private capture path. Raw
+`operator_id` and `channel_id` are retained independently of composite display
+IDs, together with source and target addresses and native port indices.
+
+Library capture defaults to topology-only (actual operator and channel creation).
+`Backend::set_observer` (`ObserverOptions {detail_full, rotate}`) sets
+`DDLOG_OBSERVER_DETAIL=full` and `DDLOG_OBSERVER_ROTATE=1` for the native child;
+managed worlds turn both on, library users leave both off. The runtime only adds the
+variables it is configured with and never strips inherited `DDLOG_OBSERVER_*`
+variables, so an operator who exports them to a library host or the MCP server still
+gets the capture they asked for. Full tracing adds runtime overhead and is not
+required by the world inventory.
+
+## Reader, state and activity
+
+`telemetry::Reader` owns a capture file position (path, offset, pending partial
+line) and is driven by a tailer thread or a one-shot pass; `telemetry::State` is
+the shared ingested view (`Arc<Mutex<State>>`) that `status` snapshots. Each tick
+reads at most 4 MiB; a single event line above 1 MiB, a malformed record or a
+conflicting operator/channel identity is a sticky error. Ingest handles `Operates`,
+`Channels`, `Schedule` (Start/Stop pairs → `schedule_count`, `busy_ns`,
+`last_seen_ns`), `Messages` (`is_send` → channel `message_count`, `records +=
+length`), `Shutdown` (`active: false`), `stream: progress` (counter),
+`stream: differential` (per-operator `arrangement_events`, `last_arrangement_event
+{kind, time_ns, length}`) and `capture_status` (`truncated`, or `rotated`, counted
+once per rotation whether the shrink or the record is observed first).
+
+`inspection.state` is `missing` (no capture file yet), `failed` (sticky error),
+`truncated`, `pending` (no operators yet) or `available` (no error, not truncated,
+operators observed). Lag never changes the state; it is reported in
+`inspection.activity`:
+
+```json
+"activity": {
+  "nodes":    {"<node id>": {"schedule_count":0,"busy_ns":0,"last_seen_ns":0,"active":true,
+                             "arrangement_events":0,"last_arrangement_event":null}},
+  "channels": {"<channel id>": {"message_count":0,"records":0}},
+  "totals":   {"events":0,"timely":0,"progress":0,"differential":0,
+               "operators":0,"channels":0,"unresolved_channels":0},
+  "last_event_ns": null, "progress_events": 0,
+  "complete": true, "truncated_at_bytes": null, "rotations": 0,
+  "lag_bytes": 0, "last_ingest_unix_ms": 0
+}
+```
+
+`complete` is whether the reader has ingested every complete line the file held at
+its last tick without error or truncation. Every observed operator and channel has
+an activity entry (zeros until events arrive); events naming an operator that never
+reported `Operates` are kept under that id as well. Channels whose endpoints are not
+known are counted in `unresolved_channels` and never emitted as edges.

docs/world-extraction.md

@@ -0,0 +1,26 @@
+# Program extraction: distinct from inspection groups
+
+The control-plane work does not turn authored display groups into independent
+programs. A future user-directed extraction action must operate on a selected
+rule/operator subgraph, infer and present its typed boundary, and register an
+immutable child program. It must rewrite the parent composition to the child's
+exact version and preserve public relation names, types and update semantics.
+
+Admission must reuse the existing parser/lowering, typed interface and composition
+validation paths. Reject cuts through registered-operation ownership, unsafe
+negation/recursion, private native state or unsupported operator boundaries. Show
+the candidate child definition, typed inputs/outputs, parent wiring and source
+provenance before publishing. Compilation/admission failure must leave the saved
+parent and any active instances unchanged.
+
+Validation must compare original and rewritten programs on user-selected regression
+traces including insertions and retractions, recursive fixed points and empty
+inputs. Example traces are evidence, not a universal proof of equivalence. When a
+structural equivalence argument cannot be established, report that limitation
+explicitly and require review of the proposed transformation. Publishing definitions
+and choosing to run the rewritten composition are separate actions.
+
+The current observer can inspect declared child blocks and recursively navigate
+exact pins. Leaf topology requires a compiled/captured native artifact; browsing
+does not compile or launch a world implicitly. No extraction mutation is currently
+exposed by the control-plane protocol.