Implementation plan: morphir-host Phase A, part 2 (native channels, probe, CLI one-shot calls)

[DamianReeves]

morphir-host Phase A, part 2: native channels, probe, and CLI one-shot calls Implementation Plan

This discussion is the implementation plan for Phase A, part 2 of the host library work. Design: #949 (checkpoints 5, 6, 10, 14 and 18). Part 1 plan and its rulings: #976. Tracking: epic #958, Phase A #959.

The body holds the overview. Each task is a separate comment below, in order.

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Put the process, Extism, and in-process native transports behind morphir_host::Channel in a new crate morphir-host-native, move the describe probe and the negotiation checks into morphir-host, and run every CLI one-shot extension call (compile, generate, workspace discovery, install probe, publish describe, configured providers) through morphir_host::Session.

Architecture: morphir-host stays portable. It gains Channel::abort, ExpectedChecks (the daemon's full negotiation rules, moved), and describe (the install probe with session fallback, moved). The new crate morphir-host-native owns everything that needs tokio processes, Extism, or non-portable validation crates: ProcessLaunch, framing, staging, ProcessChannel, ExtismChannel, NativeChannel, result validation, and activate. morphir-daemon keeps its public API and delegates to both crates. The umbrella CLI then drops its typestate one-shot paths. The workbench session pool and the provider registry stay on the daemon until part 3.

Tech Stack: Rust 2024 (rust-version = "1.98"), tokio (process, io-util, time, rt), extism 1, tempfile 3, semver 1, async-trait 0.1, morphir-extension-sdk, morphir-core, morphir-distribution, morphir-workspace.

Spec: design record #949, checkpoints 5, 6, 10, 14 and 18 (local .dev/docs/superpowers/specs/2026-09-23-morphir-host-library-design.md). Part 1 plan and its rulings: #976. Tracking: epic #958, Phase A #959, finos/morphir-rust#237 (rest), finos/morphir-rust#240 (transports part), #963 (rest).

---

Status: All tasks done. See the closing checkpoint comment.

Scope of this plan

In part 2:

• morphir-host: Channel::abort, impl Channel for Box<C>, ExpectedChecks and the expectation types, and describe.
• New crate morphir-host-native (checkpoint 18): process launch, framing and staging, ProcessChannel, NativeChannel, ExtismChannel with the Extism container, result validation, CheckedConnection, and activate.
• morphir-daemon delegates to the moved code. Its public API does not change.
• Umbrella CLI: install probe, publish describe, configured providers, installed providers (ProcessMep, WasmMep), native MEP providers, and the workbench workspace provider run through morphir_host::Session.

Part 3 gets its own plan:

• Registry and GuestSource, and provider resolution out of ExtensionRegistry (CLI Structure #238).
• Pool, and the workbench leaves sessions.rs and SessionHandle (CLI Structure #239).
• morphir-daemon drops its typestate session and actor, and removes ExtensionResponseExt (re-install npm #240 rest).

Decisions this plan makes inside the spec

1. Channel gains abort with a default that calls close. Part 1 used close for both an orderly close after exit and an abort after a protocol failure. A process channel must kill its child on abort, but wait for it on close. Waiting after a failure would hang for the full request timeout.
2. describe lives in portable morphir-host, not in morphir-host-native. It needs only the SDK and semver, so a browser host can probe too.
3. Method-result validation (compile IR checks, generated artifact paths) moves to morphir-host-native, because it needs morphir-core and morphir-distribution. CheckedConnection wraps any GuestConnection and applies it. A validation failure closes the session in order, the same way a decode failure does (part 1 ruling R6). The daemon aborted instead. Both end the session and report the error.
4. The daemon's Extism host functions (MorphirHostFunctions) are stubs today. They move unchanged. This plan does not implement them.
5. The CLI keeps NativeDirect calls (typed Rust calls on built-in providers) as they are. They are not MEP sessions.

Global Constraints

• Crates: ecosystem/morphir-rust/crates/morphir-host (portable) and ecosystem/morphir-rust/crates/morphir-host-native (new). Workspace members are crates/*. Add morphir-host-native to root default-members, to the README crate table, and to every .github/ci-impact.toml crates list that names morphir-daemon (part 1 lesson: classify.py only tests default-members).
• morphir-host must still pass cargo check -p morphir-host --target wasm32-unknown-unknown and must not depend on tokio (except as a dev-dependency), morphir-daemon, morphir-host-native, morphir-distribution, morphir-core, or any language binding.
• morphir-host-native must not depend on morphir-daemon. morphir-daemon depends on morphir-host-native.
• Every trait a browser client implements keeps the MaybeSend plus paired cfg_attr(async_trait) pattern.
• The wire sequence to released guests does not change:
  • Session: morphir.initialize (id 1), calls (ids 2, 3, ...), morphir.shutdown, morphir.exit.
  • Probe: morphir.extension.describe (id 1). Fallback: morphir.initialize (id 2), morphir.initialized notification, morphir.extension.capabilities (id 3), morphir.shutdown (id 4), then morphir.exit.
  • Frames: Content-Length: {len}\r\n\r\n{json}.
• Existing error texts stay byte-identical. The daemon and CLI tests match them.
• The CLI's own message formats stay: Provider '{provider}' rejected '{method}': {error}, Provider '{provider}' failed during {operation}: {error}, Failed to start provider '{provider}': {error}.
• Commits: Conventional Commits. No AI attribution, no Co-Authored-By.
• Doc comments in plain English. No em dashes.
• Local process tests are flaky in parallel on this machine (100 ms timeouts). Run real-process suites with --test-threads=1 when judging a failure.

Review Focus

1. An abort that waits. After a protocol failure a process channel must kill the child promptly, not wait up to 30 s. Pinned by the task 1 test a_protocol_failure_aborts_rather_than_closes and the task 5 test abort_kills_the_child_without_waiting.
2. A second exit. The daemon's process terminate sent morphir.exit itself. JsonRpcConnection::close now sends it through send, so ProcessChannel::close must not send another. Pinned by the task 5 test close_after_exit_waits_without_a_second_exit.
3. Probe wire sequence drift. Released guests (Elm 0.3.1, scala-elm) fall back to the session path with exact ids. Pinned by the task 3 test fallback_uses_the_released_ids_and_order and by the host compatibility suite in task 9.
4. Weaker validation after the move. Installed providers must still get discovery locks, legacy backend rules, and artifact path checks. Pinned by the moved daemon tests in tasks 2 and 7 and by the task 9 test that generates through an installed provider with a traversal artifact (generate_rejects_traversal_from_an_installed_provider_without_writing, already in the umbrella).
5. Staging leaks. Verified bytes are staged in private temp directories. A channel must keep the staging directory alive until the child exits, and remove it after. Pinned by the moved test verified_bytes_stage_under_the_explicit_managed_directory and the umbrella test staging_is_private_under_home_and_removed_after_success_or_failure.

---

File map

finos/morphir-rust (ecosystem/morphir-rust/):

File	Responsibility
crates/morphir-host/src/channel.rs	Channel::abort, impl Channel for Box<C>
crates/morphir-host/src/jsonrpc.rs	Use abort on failure
crates/morphir-host/src/testing.rs	MemoryChannel counts aborts
crates/morphir-host/src/expected.rs	ExpectedExtension, PersistedExtensionCapabilities, CapabilityExpectation, ExpectedChecks, validate_negotiation (moved)
crates/morphir-host/src/probe.rs	describe, Description, DescriptionSource, permits_fallback, check_capability_kinds (moved)
crates/morphir-host-native/Cargo.toml	New crate
crates/morphir-host-native/src/lib.rs	Exports
crates/morphir-host-native/src/process/{mod.rs,launch.rs,frame.rs,stage.rs,child.rs,channel.rs}	Launch, framing, staging, child, ProcessChannel
crates/morphir-host-native/src/native.rs	NativeChannel
crates/morphir-host-native/src/extism/{mod.rs,container.rs,host_functions.rs,channel.rs}	Extism container (moved), ExtismChannel
crates/morphir-host-native/src/results.rs	Result validation (moved), CheckedConnection
crates/morphir-host-native/src/activate.rs	activate, ActivatedGuest
crates/morphir-daemon/src/extensions/**	Delegate to the two crates, keep the public API

finos/morphir (umbrella):

File	Responsibility
crates/morphir/Cargo.toml	Add morphir-host-native
crates/morphir/src/extensions.rs	One-shot calls through morphir_host::Session
crates/morphir/src/extensions/guest.rs	New: open a checked connection for a resolved or configured provider, map errors to CliError
crates/morphir/src/commands/extension/probe.rs, commands/extension.rs	describe through ProcessChannel
crates/morphir/src/commands/ui/provider/extension.rs	Workspace provider through a session
crates/morphir/tests/support/process_bundle.rs	describe through ProcessChannel

Branches and pull requests

1. Tasks 1 to 7: one branch of finos/morphir-rust from origin/main in the submodule checkout (git -C ecosystem/morphir-rust switch -c feat/morphir-host-native origin/main). One PR after a whole-branch review. It refers to Adding badge for LCR interactive #237 and re-install npm #240.
2. Tasks 8 and 9: one umbrella branch from origin/main after that PR merges and main CI passes on the merge commit. Pin the submodule to the merge commit. The PR refers to feat(cli): run compile, generate and probe through morphir-host #963 and Phase A: host library in the CLI #959.

Tasks

1. Task 1: Channel::abort and boxed channels (done)
2. Task 2: Move the negotiation rules into morphir-host as ExpectedChecks (done)
3. Task 3: Move the describe probe into morphir-host (done)
4. Task 4: Create morphir-host-native with process launch, framing and staging (done)
5. Task 5: ProcessChild and ProcessChannel (done)
6. Task 6: NativeChannel and ExtismChannel (done)
7. Task 7: Result validation, CheckedConnection, and activate (done)
8. Task 8: CLI probe and publish describe through morphir_host::describe (done)
9. Task 9: CLI one-shot calls through morphir_host::Session (done)
10. After part 2

[DamianReeves]

Task 1: Channel::abort and boxed channels

Status: Done in finos/morphir-rust#263 (merged as 3b4e4fe).

Files:

• Modify: crates/morphir-host/src/channel.rs
• Modify: crates/morphir-host/src/jsonrpc.rs (abort)
• Modify: crates/morphir-host/src/testing.rs
• Modify: crates/morphir-host/tests/connection.rs

Interfaces:

• Produces: Channel::abort(&mut self) -> Result<ChannelState, ChannelError> (default: self.close().await); impl<C: Channel + ?Sized> Channel for Box<C>; SentLog::aborts() -> u32.

• Behaviour: JsonRpcConnection calls channel.abort() wherever it called channel.close() inside abort (protocol failure, failed handshake, core Failed). Orderly close still calls channel.close() after exit.

• Step 1: Write the failing tests

In crates/morphir-host/tests/connection.rs, change the two tests that pin the old behaviour and add one:

#[tokio::test]
async fn an_invalid_response_aborts_the_channel_and_fails() {
    let channel = MemoryChannel::new()
        .respond(ok(1, frontend_initialize_result("guest")))
        .respond(ok(42, json!({})));
    let log = channel.log();
    let mut connection = JsonRpcConnection::new(channel, BasicChecks::new("guest"));
    connection.open(params()).await.unwrap();

    match connection.call(methods::COMPILE, json!({})).await {
        Err(CallError::Failed(error)) => assert_eq!(
            error.to_string(),
            "Extension response ID 42 did not match request ID 2"
        ),
        other => panic!("expected a failure, got {other:?}"),
    }
    assert_eq!(log.aborts(), 1);
    assert_eq!(log.closes(), 0);
}

#[tokio::test]
async fn a_failed_handshake_aborts_the_channel() {
    let channel = MemoryChannel::new().respond(ok(1, frontend_initialize_result("impostor")));
    let log = channel.log();
    let mut connection = JsonRpcConnection::new(channel, BasicChecks::new("guest"));

    connection.open(params()).await.unwrap_err();
    assert_eq!(log.aborts(), 1);
    assert_eq!(log.closes(), 0);
}

#[tokio::test]
async fn a_protocol_failure_aborts_rather_than_closes() {
    let channel = MemoryChannel::new()
        .respond(ok(1, frontend_initialize_result("guest")))
        .respond(ok(2, json!({})))
        .respond(ok(3, json!({})));
    let log = channel.log();
    let mut connection = JsonRpcConnection::new(channel, BasicChecks::new("guest"));
    connection.open(params()).await.unwrap();
    connection.call(methods::COMPILE, json!({})).await.unwrap();
    connection.close().await.unwrap();
    assert_eq!(log.closes(), 1);
    assert_eq!(log.aborts(), 0);
}

Replace the old an_invalid_response_closes_the_channel_and_fails and a_failed_handshake_closes_the_channel with the first two. Remove the old log.closes() == 1 checks for failure paths anywhere else in the file. Keep every test that pins orderly close (closes() == 1 after ShutDown).

Add a boxed channel test:

#[tokio::test]
async fn a_boxed_channel_is_a_channel() {
    let channel: Box<dyn morphir_host::Channel> = Box::new(
        MemoryChannel::new()
            .respond(ok(1, frontend_initialize_result("guest")))
            .respond(ok(2, json!({}))),
    );
    let mut connection = JsonRpcConnection::new(channel, BasicChecks::new("guest"));
    connection.open(params()).await.unwrap();
    connection.close().await.unwrap();
}

• Step 2: Run the tests and confirm they fail

Run: cargo test -p morphir-host --test connection
Expected: compile error no method named aborts found for struct SentLog.

• Step 3: Implement

In channel.rs, add to the trait, after close:

    /// Stop the guest after a failure, without waiting for an orderly exit.
    ///
    /// The default calls [`Channel::close`]. A channel whose `close` waits for
    /// the guest to exit must override this to stop the guest at once.
    async fn abort(&mut self) -> Result<ChannelState, ChannelError> {
        self.close().await
    }

After the trait:

#[cfg_attr(not(target_arch = "wasm32"), async_trait)]
#[cfg_attr(target_arch = "wasm32", async_trait(?Send))]
impl<C: Channel + ?Sized> Channel for Box<C> {
    async fn send(&mut self, message: Outgoing) -> Result<(), ChannelError> {
        (**self).send(message).await
    }

    async fn receive(&mut self) -> Result<ExtensionResponse, ChannelError> {
        (**self).receive().await
    }

    async fn close(&mut self) -> Result<ChannelState, ChannelError> {
        (**self).close().await
    }

    async fn abort(&mut self) -> Result<ChannelState, ChannelError> {
        (**self).abort().await
    }
}

In jsonrpc.rs, inside async fn abort(&mut self, error: HostError) -> HostError, change self.channel.close().await to self.channel.abort().await. Leave close() unchanged.

In testing.rs, add aborts: Arc<Mutex<u32>> to SentLog, a method pub fn aborts(&self) -> u32, and override abort in impl Channel for MemoryChannel to increment aborts and return Ok(ChannelState::Stopped).

• Step 4: Run all host tests, lint, wasm32

Run: cargo test -p morphir-host && cargo clippy -p morphir-host --all-targets --all-features -- -D warnings && cargo check -p morphir-host --target wasm32-unknown-unknown && cargo doc -p morphir-host --no-deps
Expected: all pass, no warnings.

The daemon does not use JsonRpcConnection. Also run cargo test -p morphir-daemon --lib to confirm.

• Step 5: Commit

git add crates/morphir-host
git commit -m "feat(host): let a channel abort its guest without an orderly exit"

[DamianReeves]

Task 2: Move the negotiation rules into morphir-host as ExpectedChecks

Status: Done in finos/morphir-rust#263 (merged as 3b4e4fe).

Files:

• Create: crates/morphir-host/src/expected.rs
• Modify: crates/morphir-host/src/lib.rs
• Modify: crates/morphir-daemon/src/extensions/session/transport.rs (remove PersistedExtensionCapabilities, CapabilityExpectation, ExpectedExtension; re-export)
• Modify: crates/morphir-daemon/src/extensions/session/validation.rs (validate_negotiation, frontend_differences, persisted_frontend, backend_differences move out; DaemonChecks wraps ExpectedChecks)
• Modify: every daemon file that reads ExpectedExtension or CapabilityExpectation fields directly (find with rg -n '\.allows_legacy_backend|expected\.id|\.discovered\b|\.capabilities\b' crates/morphir-daemon/src)

Interfaces:

• Produces in morphir-host (all pub, exported from the crate root):

  • PersistedExtensionCapabilities with new(frontend: Option<FrontendCapability>, backend: Option<BackendCapability>), frontend(), backend(), is_empty().
  • CapabilityExpectation::{Exact(ExtensionCapabilities), Persisted(PersistedExtensionCapabilities), Backend(BackendCapability)}.
  • ExpectedExtension with the daemon's constructors identified, discovered, legacy_discovered, discovered_with_capabilities, discovered_with_persisted_capabilities, discovered_with_backend_capability, discovered_with_expectation, the getters id, extension_info, capabilities, persisted_capabilities, frontend_capability, backend_capability, and a new getter allows_legacy_backend(&self) -> bool plus expectation(&self) -> Option<&CapabilityExpectation>.
  • pub fn validate_negotiation(expected: ExpectedExtension, offered: &[String], result: InitializeResult) -> Result<Negotiated, HostError>.
  • pub struct ExpectedChecks with new(expected: ExpectedExtension) and impl SessionChecks for ExpectedChecks { type Error = HostError; }.

• Daemon keeps these names at the same paths through pub use morphir_host::{ExpectedExtension, PersistedExtensionCapabilities} and pub(in crate::extensions) use morphir_host::CapabilityExpectation.

• Step 1: Move the code

1. Copy PersistedExtensionCapabilities, CapabilityExpectation, and ExpectedExtension (with every constructor and getter) from crates/morphir-daemon/src/extensions/session/transport.rs into crates/morphir-host/src/expected.rs. Make every field private. Make legacy_discovered, discovered_with_expectation, and PersistedExtensionCapabilities::is_empty pub and mark legacy_discovered and discovered_with_expectation #[doc(hidden)] with a doc line: "For hosts that restore a catalog record written by an older CLI." Add:

   /// Whether a schema v1 backend may generate without a typed backend capability.
   pub fn allows_legacy_backend(&self) -> bool {
       self.allows_legacy_backend
   }
   
   /// What the host locked about the guest's capabilities, if anything.
   pub fn expectation(&self) -> Option<&CapabilityExpectation> {
       self.capabilities.as_ref()
   }

2. Copy validate_negotiation, frontend_differences, persisted_frontend, and backend_differences from crates/morphir-daemon/src/extensions/session/validation.rs into expected.rs. Change every DaemonError::Extension(text) to HostError::Invalid(text), except the offered-version check, which becomes HostError::VersionNotOffered { selected, offered: offered_versions.to_vec() }. The displayed texts must not change. Replace field reads of expected with the getters where the fields are now private (inside the same module, direct field access is fine).

3. Add:

   /// The daemon's full negotiation rules: offered version, identity, the
   /// discovery lock on metadata and capabilities, kind and capability
   /// consistency, and the schema v1 legacy backend exemption.
   #[derive(Debug, Clone)]
   pub struct ExpectedChecks {
       expected: ExpectedExtension,
   }
   
   impl ExpectedChecks {
       /// Check the guest against what the host expects of it.
       pub fn new(expected: ExpectedExtension) -> Self {
           Self { expected }
       }
   }
   
   impl SessionChecks for ExpectedChecks {
       type Error = HostError;
   
       fn negotiate(
           &mut self,
           offered: &[String],
           result: InitializeResult,
       ) -> Result<Negotiated, HostError> {
           validate_negotiation(self.expected.clone(), offered, result)
       }
   }

4. Export from lib.rs: mod expected; and pub use expected::{CapabilityExpectation, ExpectedChecks, ExpectedExtension, PersistedExtensionCapabilities, validate_negotiation};.

5. In the daemon, delete the moved definitions. In session/transport.rs add:

   pub(in crate::extensions) use morphir_host::CapabilityExpectation;
   pub use morphir_host::{ExpectedExtension, PersistedExtensionCapabilities};

   Keep session.rs and extensions/mod.rs re-exports unchanged so external paths still work.

6. In validation.rs, replace the moved functions with:

   pub(in crate::extensions) fn validate_negotiation(
       expected: ExpectedExtension,
       offered_versions: &[String],
       result: InitializeResult,
   ) -> Result<NegotiatedSession> {
       morphir_host::validate_negotiation(expected, offered_versions, result).map_err(Into::into)
   }

   Change DaemonChecks::negotiate to call it (no behaviour change). Keep DaemonChecks (its Error stays DaemonError).

7. Fix every daemon read of a now-private field (for example expected.allows_legacy_backend becomes expected.allows_legacy_backend(), expected.id becomes expected.id(), and matches on expected.capabilities become expected.expectation()).

• Step 2: Add host tests

In expected.rs, add a #[cfg(test)] mod tests that covers the moved rules through ExpectedChecks:

#[cfg(test)]
mod tests {
    use super::*;
    use morphir_extension_sdk::protocol::InitializeResult;
    use morphir_extension_sdk::{
        BackendCapability, ExtensionCapabilities, ExtensionInfo, ExtensionType, FrontendCapability,
    };

    fn info(types: Vec<ExtensionType>) -> ExtensionInfo {
        ExtensionInfo {
            id: "guest".into(),
            name: "Guest".into(),
            version: "1.0.0".into(),
            types,
            ..ExtensionInfo::default()
        }
    }

    fn result(info: ExtensionInfo, capabilities: ExtensionCapabilities) -> InitializeResult {
        InitializeResult {
            protocol_version: "0.1".into(),
            extension: info,
            capabilities,
        }
    }

    fn offered() -> Vec<String> {
        vec!["0.1".into()]
    }

    #[test]
    fn a_discovered_version_must_not_drift() {
        let mut discovered = info(vec![ExtensionType::Backend]);
        discovered.version = "2.0.0".into();
        let backend = ExtensionCapabilities {
            backend: Some(BackendCapability {
                targets: vec!["avro".into()],
                ir_versions: vec!["4".into()],
                generate: true,
            }),
            ..ExtensionCapabilities::default()
        };
        let mut checks = ExpectedChecks::new(ExpectedExtension::discovered(discovered));
        let error = checks
            .negotiate(&offered(), result(info(vec![ExtensionType::Backend]), backend))
            .unwrap_err();
        assert!(error.to_string().contains("guest"), "{error}");
    }

    #[test]
    fn a_legacy_backend_may_omit_its_capability() {
        let mut checks = ExpectedChecks::new(ExpectedExtension::legacy_discovered(info(vec![
            ExtensionType::Backend,
        ])));
        let negotiated = checks
            .negotiate(
                &offered(),
                result(info(vec![ExtensionType::Backend]), ExtensionCapabilities::default()),
            )
            .unwrap();
        assert!(negotiated.supports_method(
            morphir_extension_sdk::protocol::methods::GENERATE
        ));
    }

    #[test]
    fn an_exact_capability_lock_names_the_members_that_differ() {
        let locked = ExtensionCapabilities {
            frontend: Some(FrontendCapability {
                compile: true,
                ..FrontendCapability::default()
            }),
            ..ExtensionCapabilities::default()
        };
        let reported = ExtensionCapabilities {
            frontend: Some(FrontendCapability {
                compile: true,
                incremental: true,
                ..FrontendCapability::default()
            }),
            ..ExtensionCapabilities::default()
        };
        let mut checks = ExpectedChecks::new(ExpectedExtension::discovered_with_capabilities(
            info(vec![ExtensionType::Frontend]),
            locked,
        ));
        let error = checks
            .negotiate(&offered(), result(info(vec![ExtensionType::Frontend]), reported))
            .unwrap_err()
            .to_string();
        assert!(error.contains("incremental"), "{error}");
    }
}

Before relying on the substring asserts, read the moved validate_negotiation texts and replace each contains with the exact expected text.

• Step 3: Run the host tests and the full daemon suite

Run: cargo test -p morphir-host && cargo test -p morphir-daemon --lib --tests
Expected: all pass. The daemon's negotiation tests in session/tests.rs (for example rejects_capability_drift_from_discovery, accepts_display_name_drift_from_discovery, a_capability_mismatch_names_the_members_that_differ, explicit_schema_v1_discovery_retains_legacy_generation) and activation/tests.rs must pass unchanged. That proves the texts and rules did not change.

Run: cargo clippy -p morphir-host -p morphir-daemon --all-targets --all-features -- -D warnings && cargo check -p morphir-host --target wasm32-unknown-unknown

• Step 4: Commit

git add crates/morphir-host crates/morphir-daemon
git commit -m "refactor(host): move the negotiation rules into morphir-host as ExpectedChecks"

[DamianReeves]

Task 3: Move the describe probe into morphir-host

Status: Done in finos/morphir-rust#263 (merged as 3b4e4fe).

Files:

• Create: crates/morphir-host/src/probe.rs
• Modify: crates/morphir-host/Cargo.toml (add semver = "1")
• Modify: crates/morphir-host/src/lib.rs
• Test: in probe.rs (#[cfg(test)], with tokio::test)

Interfaces:

• Produces:

  • pub enum DescriptionSource { Describe, SessionFallback } (Debug, Clone, Copy, PartialEq, Eq)
  • pub struct Description { pub claims: CapabilityClaimSet, pub source: DescriptionSource } (Debug, Clone)
  • pub async fn describe<C: Channel>(channel: C, config: &HostConfig, expected_id: &str) -> Result<Description, HostError>

• describe owns the channel. On success it sends morphir.exit and closes the channel. On any failure it aborts the channel and returns the error. An Indeterminate close after success is an error: HostError::Channel { message: "Extension shutdown outcome is indeterminate".into(), state: ChannelState::Indeterminate }.

• Fallback negotiation uses BasicChecks::new(expected_id) (identity, offered version, kind and capability consistency). That is the same rule set the daemon applied, because the probe launch carried only an identity.

• Step 1: Write the failing tests

crates/morphir-host/src/probe.rs (tests first; the module body follows in step 3):

#[cfg(test)]
mod tests {
    use super::*;
    use crate::testing::{MemoryChannel, frontend_initialize_result};
    use morphir_extension_sdk::claims::CapabilityClaimSet;
    use morphir_extension_sdk::protocol::{
        ExtensionResponse, PeerInfo, PeerKind, RpcError, error_codes, methods,
    };
    use serde_json::json;

    fn config() -> HostConfig {
        HostConfig::new(PeerInfo {
            kind: PeerKind::Unspecified,
            name: "test".into(),
            version: "1.0.0".into(),
        })
    }

    fn claims(id: &str) -> CapabilityClaimSet {
        let init = frontend_initialize_result(id);
        CapabilityClaimSet::from_metadata(
            vec![init.protocol_version],
            init.extension,
            &init.capabilities,
        )
        .unwrap()
    }

    fn ok(id: u64, value: impl serde::Serialize) -> ExtensionResponse {
        ExtensionResponse::success(id, value).unwrap()
    }

    fn rpc(id: u64, code: i32, message: &str) -> ExtensionResponse {
        ExtensionResponse::error(
            id,
            RpcError {
                code,
                message: message.into(),
                data: None,
            },
        )
    }

    #[tokio::test]
    async fn a_direct_description_is_returned_and_the_guest_exits() {
        let channel = MemoryChannel::new().respond(ok(1, claims("guest")));
        let log = channel.log();
        let description = describe(channel, &config(), "guest").await.unwrap();
        assert_eq!(description.source, DescriptionSource::Describe);
        assert_eq!(description.claims.extension.id, "guest");
        assert_eq!(log.methods(), [methods::DESCRIBE, methods::EXIT]);
        assert_eq!(log.closes(), 1);
    }

    #[tokio::test]
    async fn fallback_uses_the_released_ids_and_order() {
        let channel = MemoryChannel::new()
            .respond(rpc(1, error_codes::METHOD_NOT_FOUND, "no"))
            .respond(ok(2, frontend_initialize_result("guest")))
            .respond(ok(3, json!({"frontend": {"languages": [], "irVersions": [], "compile": true, "incremental": false, "fragments": false}})))
            .respond(ok(4, json!({})));
        let log = channel.log();
        let description = describe(channel, &config(), "guest").await.unwrap();
        assert_eq!(description.source, DescriptionSource::SessionFallback);
        assert_eq!(
            log.methods(),
            [
                methods::DESCRIBE,
                methods::INITIALIZE,
                methods::INITIALIZED,
                methods::CAPABILITIES,
                methods::SHUTDOWN,
                methods::EXIT
            ]
        );
        assert_eq!(log.ids(), [1, 2, 3, 4]);
    }

    #[tokio::test]
    async fn a_legacy_refusal_message_falls_back() {
        let channel = MemoryChannel::new()
            .respond(rpc(1, error_codes::INVALID_REQUEST, "Extension is not initialized"))
            .respond(ok(2, frontend_initialize_result("guest")))
            .respond(ok(3, json!({})))
            .respond(ok(4, json!({})));
        let description = describe(channel, &config(), "guest").await.unwrap();
        assert_eq!(description.source, DescriptionSource::SessionFallback);
    }

    #[tokio::test]
    async fn an_internal_error_does_not_fall_back_and_aborts() {
        let channel = MemoryChannel::new().respond(rpc(1, error_codes::INTERNAL_ERROR, "boom"));
        let log = channel.log();
        let error = describe(channel, &config(), "guest").await.unwrap_err();
        assert_eq!(error.to_string(), "RPC error -32603: boom");
        assert_eq!(log.aborts(), 1);
        assert_eq!(log.methods(), [methods::DESCRIBE]);
    }

    #[tokio::test]
    async fn a_different_identity_is_refused() {
        let channel = MemoryChannel::new().respond(ok(1, claims("impostor")));
        let error = describe(channel, &config(), "guest").await.unwrap_err();
        assert_eq!(
            error.to_string(),
            "Description extension identity differs from launch identity"
        );
    }
}

frontend_initialize_result declares a frontend with a frontend capability, so the fallback capability map in a_legacy_refusal_message_falls_back is not checked against kinds (a fallback claim set comes from a validated session, as in the daemon). If CapabilityClaimSet::from_session requires a non-empty map, use the same map as the first fallback test.

Add SentLog::ids(&self) -> Vec<u64> to testing.rs (ids of requests, in order, notifications skipped).

• Step 2: Run the tests and confirm they fail

Run: cargo test -p morphir-host probe
Expected: compile error cannot find function describe.

• Step 3: Implement probe.rs

Move permits_fallback and check_capability_kinds from crates/morphir-daemon/src/extensions/process/describe.rs into probe.rs, changing DaemonError::Extension(text) to HostError::Invalid(text) with identical text. Then:

//! The install probe: describe a guest before trusting its claims.

use crate::channel::{Channel, Outgoing};
use crate::envelope::{EnvelopeError, validate_envelope};
use crate::session_core::{BasicChecks, SessionChecks};
use crate::{ChannelState, HostConfig, HostError};
use morphir_extension_sdk::claims::CapabilityClaimSet;
use morphir_extension_sdk::protocol::{
    DescribeParams, ExtensionNotification, ExtensionRequest, ExtensionResponse, InitializeResult,
    RpcError, error_codes, methods,
};
use morphir_extension_sdk::{ExtensionCapabilities, ExtensionType};
use serde_json::{Map, Value};

/// How the host obtained a guest's claim set.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum DescriptionSource {
    /// The guest answered `morphir.extension.describe` directly.
    Describe,
    /// The host rebuilt the claim set from a negotiated session.
    SessionFallback,
}

/// A guest's claim set and how the host obtained it.
#[derive(Debug, Clone)]
pub struct Description {
    /// The claim set.
    pub claims: CapabilityClaimSet,
    /// How the host obtained it.
    pub source: DescriptionSource,
}

/// Describe the guest behind `channel`, then release it.
///
/// A guest that refuses `morphir.extension.describe` before initialization
/// is described through a short session instead. The ids and the order of
/// messages match what released guests were tested with.
pub async fn describe<C: Channel>(
    mut channel: C,
    config: &HostConfig,
    expected_id: &str,
) -> Result<Description, HostError> {
    match describe_inner(&mut channel, config, expected_id).await {
        Ok(description) => {
            channel
                .send(Outgoing::Notification(ExtensionNotification::without_params(
                    methods::EXIT,
                )))
                .await?;
            match channel.close().await? {
                ChannelState::Stopped => Ok(description),
                ChannelState::Indeterminate => Err(HostError::Channel {
                    message: "Extension shutdown outcome is indeterminate".into(),
                    state: ChannelState::Indeterminate,
                }),
            }
        }
        Err(Failure::Channel(error)) => Err(error),
        Err(Failure::Protocol(error)) => match channel.abort().await {
            Ok(_) => Err(error),
            Err(abort) => Err(HostError::Channel {
                message: format!("{error}; transport abort also failed: {}", abort.message),
                state: abort.state,
            }),
        },
    }
}

/// Why the probe stopped. A channel failure already released the guest.
enum Failure {
    Channel(HostError),
    Protocol(HostError),
}

impl From<HostError> for Failure {
    fn from(error: HostError) -> Self {
        Failure::Protocol(error)
    }
}

async fn exchange<C: Channel>(
    channel: &mut C,
    id: u64,
    method: &str,
    params: impl serde::Serialize,
) -> Result<ExtensionResponse, Failure> {
    let request = ExtensionRequest::new(method, params, id).map_err(HostError::from)?;
    channel
        .send(Outgoing::Request(request))
        .await
        .map_err(|error| Failure::Channel(error.into()))?;
    channel
        .receive()
        .await
        .map_err(|error| Failure::Channel(error.into()))
}

fn result<T: serde::de::DeserializeOwned>(
    response: ExtensionResponse,
    id: u64,
) -> Result<T, Failure> {
    match validate_envelope(response, id) {
        Ok(value) => Ok(serde_json::from_value(value).map_err(HostError::from)?),
        Err(EnvelopeError::Rpc(error) | EnvelopeError::Invalid(error)) => Err(error.into()),
    }
}

async fn describe_inner<C: Channel>(
    channel: &mut C,
    config: &HostConfig,
    expected_id: &str,
) -> Result<Description, Failure> {
    let offered = config.protocol_versions().to_vec();
    let response = exchange(
        channel,
        1,
        methods::DESCRIBE,
        DescribeParams {
            protocol_versions: offered.clone(),
        },
    )
    .await?;
    if response.id == 1 && response.error.as_ref().is_some_and(permits_fallback) {
        return describe_through_session(channel, config, expected_id).await;
    }
    let claims: CapabilityClaimSet = result(response, 1)?;
    if claims.extension.id != expected_id {
        return Err(HostError::Invalid(
            "Description extension identity differs from launch identity".into(),
        )
        .into());
    }
    check_capability_kinds(&claims)?;
    if !claims
        .protocol_versions
        .iter()
        .any(|version| offered.contains(version))
    {
        return Err(HostError::Invalid(
            "Description has no protocol version in common with the host".into(),
        )
        .into());
    }
    if claims
        .requires
        .as_ref()
        .is_some_and(|requirements| !requirements.host.is_empty())
    {
        let host = config.peer().version.parse().map_err(|error| {
            HostError::Invalid(format!("Invalid host SemVer for requires.host: {error}"))
        })?;
        claims
            .check_host(&host)
            .map_err(|error| HostError::Invalid(error.to_string()))?;
    }
    Ok(Description {
        claims,
        source: DescriptionSource::Describe,
    })
}

async fn describe_through_session<C: Channel>(
    channel: &mut C,
    config: &HostConfig,
    expected_id: &str,
) -> Result<Description, Failure> {
    let params = config.initialize_params();
    let initialized: InitializeResult =
        result(exchange(channel, 2, methods::INITIALIZE, &params).await?, 2)?;
    BasicChecks::new(expected_id).negotiate(&params.protocol_versions, initialized.clone())?;
    channel
        .send(Outgoing::Notification(ExtensionNotification::without_params(
            methods::INITIALIZED,
        )))
        .await
        .map_err(|error| Failure::Channel(error.into()))?;
    let capabilities: Map<String, Value> = result(
        exchange(channel, 3, methods::CAPABILITIES, serde_json::json!({})).await?,
        3,
    )?;
    let _: Value = result(
        exchange(channel, 4, methods::SHUTDOWN, serde_json::json!({})).await?,
        4,
    )?;
    Ok(Description {
        claims: CapabilityClaimSet::from_session(
            vec![initialized.protocol_version],
            initialized.extension,
            capabilities,
        ),
        source: DescriptionSource::SessionFallback,
    })
}

Paste the moved permits_fallback(error: &RpcError) -> bool and check_capability_kinds(claims: &CapabilityClaimSet) -> Result<(), HostError> below. Move the daemon's three unit tests for check_capability_kinds (declared_kinds_and_capability_objects_must_match, repeated_kinds_and_malformed_objects_are_refused, unknown_capability_members_are_not_kinds) into this module's tests.

The daemon compared response.error before checking the envelope id. response.id == 1 keeps the same effect (a mismatched id is then caught by result).

Export: mod probe; and pub use probe::{Description, DescriptionSource, describe};.

• Step 4: Point the daemon's describe at the host

In crates/morphir-daemon/src/extensions/process/describe.rs, keep SpawnedProcessTransport::describe(self, params: InitializeParams) -> Result<ProcessDescription> as the public signature, and keep DescriptionSource and ProcessDescription as daemon types. Leave its body as it is in this task: the daemon transport is not a Channel yet. Task 5 rewires it. Delete only the moved permits_fallback and check_capability_kinds, and call morphir_host's versions (make them pub and #[doc(hidden)] in morphir-host, or re-export them through a probe submodule).

• Step 5: Run the tests

Run: cargo test -p morphir-host && cargo test -p morphir-daemon --lib && cargo clippy -p morphir-host -p morphir-daemon --all-targets --all-features -- -D warnings && cargo check -p morphir-host --target wasm32-unknown-unknown
Expected: all pass.

Run the real describe fixtures: cargo build -p morphir-daemon --example mep-native-backend && MEP_NATIVE_FIXTURE=target/debug/examples/mep-native-backend cargo test -p morphir-daemon --test process_describe -- --ignored --test-threads=1
Expected: 6 passed.

• Step 6: Commit

git add crates/morphir-host crates/morphir-daemon Cargo.lock
git commit -m "feat(host): move the describe probe into morphir-host"

[DamianReeves]

Task 4: Create morphir-host-native with process launch, framing and staging

Status: Done in finos/morphir-rust#263 (merged as 3b4e4fe).

Files:

• Create: crates/morphir-host-native/Cargo.toml, src/lib.rs, src/process/mod.rs, src/process/launch.rs, src/process/frame.rs, src/process/stage.rs
• Modify: root Cargo.toml (default-members), README.md (crate table), .github/ci-impact.toml
• Modify: crates/morphir-daemon/Cargo.toml, crates/morphir-daemon/src/extensions/process.rs, process/launch.rs (deleted), process/tests.rs

Interfaces:

• Produces in morphir_host_native::process:

  • ProcessLaunch (moved unchanged, all constructors and builders) plus pub fn expectation(&self) -> ExpectedExtension (the mapping from the daemon's SpawnedProcessSession::expected_extension, lines 166 to 180) and read accessors the daemon needs: extension_id(), args(), working_directory(), environment(), request_timeout(), program().
  • pub enum ProcessProgram (moved; pub with #[doc(hidden)]).
  • pub async fn write_frame<W, T>(writer: &mut W, message: &T) -> Result<(), HostError> and pub async fn read_frame<R>(reader: &mut R) -> Result<Vec<u8>, HostError> (moved; same frame format and error texts, as HostError::Invalid or HostError::Json; I/O errors as HostError::Invalid(error.to_string())).
  • pub const MAX_MEP_PAYLOAD_BYTES: u32 = 64 * 1024 * 1024;
  • pub async fn prepare_program(program: &ProcessProgram) -> Result<(PathBuf, Option<tempfile::TempDir>), HostError> (moved with stage_verified_program, validate_verified_program_filename, make_owner_executable).

• The daemon re-exports pub use morphir_host_native::process::ProcessLaunch; at extensions::process::ProcessLaunch and extensions::ProcessLaunch. MAX_MEP_PAYLOAD_BYTES in extensions::protocol becomes pub use morphir_host_native::process::MAX_MEP_PAYLOAD_BYTES;.

• Step 1: Create the crate

crates/morphir-host-native/Cargo.toml:

[package]
name = "morphir-host-native"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
authors.workspace = true
license.workspace = true
repository.workspace = true
description = "Native channels for the Morphir extension host: processes, Extism and in-process guests"

[dependencies]
morphir-host = { path = "../morphir-host" }
morphir-extension-sdk = { path = "../morphir-extension-sdk" }
async-trait = "0.1"
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
tempfile = "3"
thiserror = { workspace = true }
tokio = { version = "1", features = ["rt", "sync", "io-util", "process", "time", "macros"] }
tracing = "0.1"

[dev-dependencies]
morphir-host = { path = "../morphir-host", features = ["testing"] }
tokio = { version = "1", features = ["rt-multi-thread", "macros", "test-util"] }

src/lib.rs:

//! Native channels for the Morphir extension host.
//!
//! `morphir-host` is portable and has no native dependencies. This crate
//! holds the parts that need a native runtime: child processes, Extism, and
//! in-process Rust guests, each behind `morphir_host::Channel`.

pub mod process;

Add "crates/morphir-host-native", to root default-members (alphabetical, after morphir-host), a README crate table row (| morphir-host-native | Native channels for the Morphir extension host: processes, Extism and in-process guests. |), and "morphir-host-native" to every .github/ci-impact.toml crates list that contains "morphir-daemon".

• Step 2: Move the code

1. git mv crates/morphir-daemon/src/extensions/process/launch.rs crates/morphir-host-native/src/process/launch.rs. Change pub(super) fields to private and add the accessors listed above. Change CapabilityExpectation and ExpectedExtension imports to morphir_host::{CapabilityExpectation, ExpectedExtension, PersistedExtensionCapabilities}. Make from_legacy_verified_bytes_in pub with #[doc(hidden)] (the daemon calls it). Add pub fn expectation(&self) -> ExpectedExtension, copying the mapping from SpawnedProcessSession::expected_extension.
2. Move write_frame and read_frame from process.rs into frame.rs, and the staging functions into stage.rs. Move MAX_MEP_PAYLOAD_BYTES into frame.rs.
3. Move the unit tests that exercise only moved code from crates/morphir-daemon/src/extensions/process/tests.rs: content_length_frames_round_trip_formatted_json, stdout_logs_are_rejected_as_protocol_headers, verified_bytes_stage_under_the_explicit_managed_directory, verified_bytes_reject_path_components_without_writing_outside_staging, discovered_process_launch_retains_exact_negotiation_metadata (change it to assert launch.expectation()).
4. src/process/mod.rs declares the modules and re-exports the public items.
5. In the daemon, add morphir-host-native = { path = "../morphir-host-native" } and replace the moved definitions with use/pub use of morphir_host_native::process::* items. Map HostError to DaemonError with the existing From impl. Where the daemon read launch.program etc., use the new accessors.

• Step 3: Run the tests

Run: cargo test -p morphir-host-native && cargo test -p morphir-daemon --lib --tests && cargo clippy -p morphir-host-native -p morphir-daemon --all-targets --all-features -- -D warnings && cargo fmt --all -- --check
Expected: all pass. The moved tests pass in the new crate.

Run: mise run test:process -- --test-threads=1 (or the MEP_NATIVE_FIXTURE command from task 3 with --test spawned_process_extension). Expected: 8 passed.

• Step 4: Commit

git add crates/morphir-host-native crates/morphir-daemon Cargo.toml Cargo.lock README.md .github/ci-impact.toml
git commit -m "feat(host-native): move process launch, framing and staging into morphir-host-native"

[DamianReeves]

Task 5: ProcessChild and ProcessChannel

Status: Done in finos/morphir-rust#263 (merged as 3b4e4fe).

Files:

• Create: crates/morphir-host-native/src/process/child.rs, src/process/channel.rs
• Modify: crates/morphir-daemon/src/extensions/process.rs, process/transport.rs, process/describe.rs
• Test: crates/morphir-host-native/tests/process_channel.rs (unix only, shell-script guests)

Interfaces:

• Produces in morphir_host_native::process:

  • pub struct ProcessChild with pub async fn spawn(launch: &ProcessLaunch) -> Result<Self, HostError>, pub fn request_timeout(&self) -> Duration, pub async fn write<T: Serialize>(&mut self, message: &T) -> Result<(), HostError> (frame under the timeout), pub async fn read(&mut self) -> Result<Vec<u8>, HostError> (frame under the timeout), pub async fn abort(&mut self) -> Result<(), HostError> (close stdin, kill if running, wait, cancel the stderr reader), pub async fn wait_for_exit(&mut self) -> Result<ExitStatus, HostError> (close stdin, wait under the timeout, collect stderr), pub fn stderr_output(&self) -> &str, pub fn is_running(&mut self) -> Result<bool, HostError>. It owns the child, stdin, stdout, the bounded stderr tail (256 KiB) and the staging TempDir. Error texts are copied from the daemon's process.rs and process/transport.rs unchanged.
  • pub struct ProcessChannel with pub async fn spawn(launch: ProcessLaunch) -> Result<Self, HostError>, pub fn expectation(&self) -> ExpectedExtension, pub fn stderr_output(&self) -> &str, and impl Channel:
    • send(Request | Notification): child.write. On error: child.abort(), then return ChannelError { message, state: Stopped }, or { "{error}; process cleanup also failed: {cleanup}", Indeterminate } when the abort fails.
    • receive(): child.read(), then parse ExtensionResponse. Same error mapping.
    • close(): does NOT send exit (the connection sent it). child.wait_for_exit(). A timeout aborts with the same mapping and the text "Extension process did not exit after {:?}". A non-zero status returns Err(ChannelError { "Extension process exited with status {status}", Stopped }). Otherwise Ok(Stopped).
    • abort(): child.abort(). Ok(Stopped) or Err(Indeterminate).

• The daemon's SpawnedProcessSession and SpawnedProcessTransport keep their public API and are rebuilt on ProcessChild. SpawnedProcessTransport::terminate keeps sending exit itself (it is not a Channel).

• SpawnedProcessTransport::describe becomes: build a ProcessChannel from the same child is not possible, so rebuild describe as morphir_host::describe(TransportChannel(self), &config, id) where TransportChannel is a private daemon adapter that implements Channel over SpawnedProcessTransport (send(Request) stores the request, receive calls exchange, send(Notification) writes the frame, close waits for exit without sending exit again, abort calls abort). Map morphir_host::Description to the daemon's ProcessDescription. The daemon method signature does not change.

• Step 1: Write the failing channel tests

crates/morphir-host-native/tests/process_channel.rs:

#![cfg(unix)]

use morphir_extension_sdk::protocol::{
    ExtensionNotification, ExtensionRequest, methods,
};
use morphir_host::{Channel, ChannelState, Outgoing};
use morphir_host_native::process::{ProcessChannel, ProcessLaunch};
use std::os::unix::fs::PermissionsExt;
use std::time::{Duration, Instant};

/// A guest that answers every frame with a success result for id 1, then
/// exits 0 when stdin closes.
const ECHO: &str = r#"#!/bin/sh
while IFS= read -r header; do
  case "$header" in
    Content-Length:*) len=${header#Content-Length: }; len=$(printf '%s' "$len" | tr -d '\r');;
    $'\r'|'') body=$(dd bs=1 count="$len" 2>/dev/null)
       case "$body" in
         *'"id"'*) out='{"jsonrpc":"2.0","result":{},"id":1}'
                   printf 'Content-Length: %s\r\n\r\n%s' "${#out}" "$out";;
       esac;;
  esac
done
exit 0
"#;

/// A guest that never exits on its own.
const HANG: &str = "#!/bin/sh\nwhile true; do sleep 1; done\n";

fn guest(dir: &tempfile::TempDir, script: &str) -> std::path::PathBuf {
    let path = dir.path().join("guest.sh");
    std::fs::write(&path, script).unwrap();
    std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o700)).unwrap();
    path
}

#[tokio::test]
async fn close_after_exit_waits_without_a_second_exit() {
    let dir = tempfile::tempdir().unwrap();
    let launch = ProcessLaunch::new("guest", guest(&dir, ECHO), dir.path())
        .request_timeout(Duration::from_secs(5));
    let mut channel = ProcessChannel::spawn(launch).await.unwrap();
    channel
        .send(Outgoing::Request(
            ExtensionRequest::new(methods::PING, serde_json::json!({}), 1).unwrap(),
        ))
        .await
        .unwrap();
    channel.receive().await.unwrap();
    channel
        .send(Outgoing::Notification(ExtensionNotification::without_params(methods::EXIT)))
        .await
        .unwrap();
    assert_eq!(channel.close().await.unwrap(), ChannelState::Stopped);
}

#[tokio::test]
async fn abort_kills_the_child_without_waiting() {
    let dir = tempfile::tempdir().unwrap();
    let launch = ProcessLaunch::new("guest", guest(&dir, HANG), dir.path())
        .request_timeout(Duration::from_secs(30));
    let mut channel = ProcessChannel::spawn(launch).await.unwrap();
    let started = Instant::now();
    assert_eq!(channel.abort().await.unwrap(), ChannelState::Stopped);
    assert!(started.elapsed() < Duration::from_secs(5), "{:?}", started.elapsed());
}

#[tokio::test]
async fn a_receive_timeout_kills_the_child_and_reports_stopped() {
    let dir = tempfile::tempdir().unwrap();
    let launch = ProcessLaunch::new("guest", guest(&dir, HANG), dir.path())
        .request_timeout(Duration::from_millis(300));
    let mut channel = ProcessChannel::spawn(launch).await.unwrap();
    channel
        .send(Outgoing::Request(
            ExtensionRequest::new(methods::PING, serde_json::json!({}), 1).unwrap(),
        ))
        .await
        .unwrap();
    let error = channel.receive().await.unwrap_err();
    assert_eq!(error.state, ChannelState::Stopped);
}

If the ECHO script is unreliable on the CI shells (dd byte counts or $'\r' handling), replace it with a small Rust example binary in crates/morphir-host-native/examples/echo-guest.rs that reuses read_frame/write_frame, and build it in the test with env!("CARGO_BIN_EXE_...") or an examples path, the way morphir-daemon builds mep-native-backend. Record which one you used.

• Step 2: Run the tests and confirm they fail

Run: cargo test -p morphir-host-native --test process_channel
Expected: compile error no ProcessChannel in process.

• Step 3: Implement ProcessChild and ProcessChannel

Move the child management from crates/morphir-daemon/src/extensions/process.rs (spawn body, spawn_child with the busy-executable retry, read_bounded_tail, append_bounded_tail, collect_stderr, abort_process, validate_launch, and the constants) into child.rs as ProcessChild. Keep the environment clearing (env_clear().envs(launch.environment())) and kill_on_drop(true). Implement ProcessChannel in channel.rs with the error mapping above. Move stderr_capture_retains_only_the_bounded_tail into this crate.

Rebuild the daemon's SpawnedProcessSession and SpawnedProcessTransport on ProcessChild, keeping every public method and error text. Rewire SpawnedProcessTransport::describe through morphir_host::describe and the private TransportChannel adapter. Delete the daemon's own describe_inner and describe_through_session.

• Step 4: Run all suites

Run: cargo test -p morphir-host-native && cargo test -p morphir-host && cargo test -p morphir-daemon --lib --tests
Run: cargo build -p morphir-daemon --example mep-native-backend && MEP_NATIVE_FIXTURE=target/debug/examples/mep-native-backend cargo test -p morphir-daemon --test spawned_process_extension --test process_describe -- --ignored --test-threads=1
Expected: all pass (8 and 6 ignored tests pass).
Run: cargo clippy -p morphir-host-native -p morphir-daemon --all-targets --all-features -- -D warnings && cargo fmt --all -- --check

• Step 5: Commit

git add crates/morphir-host-native crates/morphir-daemon Cargo.lock
git commit -m "feat(host-native): add ProcessChannel and run the daemon's process transport on it"

[DamianReeves]

Task 6: NativeChannel and ExtismChannel

Status: Done in finos/morphir-rust#263 (merged as 3b4e4fe).

Files:

• Create: crates/morphir-host-native/src/native.rs, src/extism/mod.rs, src/extism/channel.rs
• Move: crates/morphir-daemon/src/extensions/container.rs to crates/morphir-host-native/src/extism/container.rs; crates/morphir-daemon/src/extensions/host_functions.rs to crates/morphir-host-native/src/extism/host_functions.rs
• Modify: crates/morphir-host-native/Cargo.toml (add extism = "1"; dev wat = "1"), src/lib.rs
• Modify: daemon extensions/mod.rs, session/extism.rs, session/native.rs (re-export and use moved types)

Interfaces:

• Produces:

  • morphir_host_native::NativeChannel with pub fn new(extension: &NativeExtension) -> Self (uses extension.open_protocol()), pub fn expectation(&self) -> ExpectedExtension (discovered_with_capabilities(info, capabilities)), and impl Channel: send(Request) queues spawn_blocking(protocol.handle(request)); receive awaits it (a join error is Indeterminate with "Native extension protocol worker failed: {error}"); send(Notification) is a no-op (the SDK protocol takes only requests); close and abort mark the channel stopped and return Stopped; any use after that is "Native extension transport is stopped" with Stopped.
  • morphir_host_native::extism::{ExtensionContainer, ExtensionContainerBuilder, MorphirHostFunctions, MorphirHostState} (moved, unchanged APIs).
  • morphir_host_native::extism::ExtismChannel with pub fn new(container: ExtensionContainer, expected: ExpectedExtension) -> Self, pub fn expectation(&self) -> ExpectedExtension, and impl Channel: send(Request) serializes and calls container.call_raw("handle", &bytes), storing the response bytes; receive deserializes them; any error is Indeterminate (as the daemon's ExtismTransport does); send(Notification) is a no-op; close and abort return Stopped.

• The daemon keeps ExtensionContainer, MorphirHostFunctions and friends at their old paths through pub use.

• Step 1: Write the failing tests

crates/morphir-host-native/tests/native_channel.rs:

use morphir_extension_sdk::doc_fixtures::DocFrontend;
use morphir_extension_sdk::protocol::{PeerInfo, PeerKind};
use morphir_extension_sdk::{CompileRequest, NativeExtension};
use morphir_host::{ExpectedChecks, HostConfig, JsonRpcConnection, Session};
use morphir_host_native::NativeChannel;

fn config() -> HostConfig {
    HostConfig::new(PeerInfo {
        kind: PeerKind::Unspecified,
        name: "test".into(),
        version: "1.0.0".into(),
    })
}

#[tokio::test(flavor = "multi_thread")]
async fn a_native_frontend_compiles_through_a_session() {
    let extension = NativeExtension::frontend_only(DocFrontend::default()).unwrap();
    let channel = NativeChannel::new(&extension);
    let checks = ExpectedChecks::new(channel.expectation());
    let connection = JsonRpcConnection::new(channel, checks);
    let mut session = Session::open(connection, &config()).await.unwrap();
    assert_eq!(session.negotiated().extension().id, extension.info().id);
    let result = session.compile(CompileRequest::default()).await;
    session.close().await.unwrap();
    assert!(result.is_ok() || matches!(result, Err(morphir_host::CallError::Rejected(_))));
}

Check how DocFrontend is constructed in crates/morphir-extension-sdk/tests/lifecycle.rs and copy that form; if CompileRequest::default() is rejected by DocFrontend, assert the exact rejection instead of the loose match, and keep the session open and closed cleanly.

Move the container tests with the container (they use wat guests and are self-contained).

• Step 2: Run the tests and confirm they fail

Run: cargo test -p morphir-host-native --test native_channel
Expected: compile error no NativeChannel in the root.

• Step 3: Implement and move

Implement native.rs and extism/channel.rs as specified. git mv the two daemon files, fix imports, and add pub use lines in the daemon so every old path still resolves. DaemonError in the moved container becomes HostError (container errors keep their texts); the daemon maps them with From.

• Step 4: Run all suites

Run: cargo test -p morphir-host-native && cargo test -p morphir-daemon --lib --tests
Run: mise run test:extism (builds the WASM guests; needs the wasm32-unknown-unknown target).
Expected: all pass.
Run: cargo clippy -p morphir-host-native -p morphir-daemon --all-targets --all-features -- -D warnings && cargo fmt --all -- --check

• Step 5: Commit

git add crates/morphir-host-native crates/morphir-daemon Cargo.lock
git commit -m "feat(host-native): add native and Extism channels and move the Extism container"

[DamianReeves]

Task 7: Result validation, CheckedConnection, and activate

Status: Done in finos/morphir-rust#263 (merged as 3b4e4fe).

Files:

• Create: crates/morphir-host-native/src/results.rs, src/activate.rs
• Modify: crates/morphir-host-native/Cargo.toml (add morphir-core, morphir-distribution, morphir-workspace, base64 = "0.23", unicode-casefold, unicode-normalization (workspace))
• Modify: daemon session/validation.rs (result validation moves out; the daemon calls it), activation.rs (built on activate parts where it helps; public API unchanged)

Interfaces:

• Produces:

  • pub async fn validate_result(method: &str, params: serde_json::Value, value: serde_json::Value) -> Result<serde_json::Value, HostError> (moved validate_method_result_async with its helpers; texts unchanged, as HostError::Invalid).
  • pub struct CheckedConnection<G> with pub fn new(inner: G) -> Self and impl<G: GuestConnection> GuestConnection for CheckedConnection<G>: open and close delegate; call delegates, then validate_result(method, params, value). On a validation failure it closes the inner connection in order and returns CallError::Failed, combining the close error as "{error}; orderly shutdown also failed: {close}" when close fails.
  • pub struct ActivatedGuest { pub connection: CheckedConnection<JsonRpcConnection<Box<dyn Channel>, ExpectedChecks>>, pub id: String } and pub async fn activate(artifact: VerifiedExtensionArtifact, working_directory: &Path) -> Result<ActivatedGuest, HostError>. It mirrors the daemon's activate_transport: a process artifact uses ProcessLaunch::from_verified_bytes_with_persisted_capabilities_in or from_legacy_verified_bytes_in plus args, then ProcessChannel; a WASM artifact uses ExtensionContainer::from_bytes_async with MorphirHostFunctions::for_restricted_generation(working_directory) and ExtismChannel with the persisted or legacy expectation. The checks are ExpectedChecks::new(channel.expectation()).

• Step 1: Write the failing tests

crates/morphir-host-native/tests/checked_connection.rs:

use morphir_extension_sdk::protocol::{ExtensionResponse, PeerInfo, PeerKind, methods};
use morphir_extension_sdk::{BackendCapability, ExtensionCapabilities, ExtensionInfo, ExtensionType, GenerateRequest};
use morphir_extension_sdk::protocol::InitializeResult;
use morphir_host::testing::MemoryChannel;
use morphir_host::{BasicChecks, CallError, HostConfig, JsonRpcConnection, Session};
use morphir_host_native::CheckedConnection;
use serde_json::json;

fn config() -> HostConfig {
    HostConfig::new(PeerInfo { kind: PeerKind::Unspecified, name: "test".into(), version: "1.0.0".into() })
}

fn backend() -> InitializeResult {
    InitializeResult {
        protocol_version: "0.1".into(),
        extension: ExtensionInfo { id: "guest".into(), name: "Guest".into(), version: "1.0.0".into(), types: vec![ExtensionType::Backend], ..ExtensionInfo::default() },
        capabilities: ExtensionCapabilities {
            backend: Some(BackendCapability { targets: vec!["x".into()], ir_versions: vec!["4".into()], generate: true }),
            ..ExtensionCapabilities::default()
        },
    }
}

#[tokio::test(flavor = "multi_thread")]
async fn a_traversal_artifact_ends_the_session() {
    let channel = MemoryChannel::new()
        .respond(ExtensionResponse::success(1, backend()).unwrap())
        .respond(ExtensionResponse::success(2, json!({
            "success": true,
            "artifacts": [{"path": "../escape.txt", "content": "x", "binary": false}],
            "diagnostics": []
        })).unwrap())
        .respond(ExtensionResponse::success(3, json!({})).unwrap());
    let log = channel.log();
    let connection = CheckedConnection::new(JsonRpcConnection::new(channel, BasicChecks::new("guest")));
    let mut session = Session::open(connection, &config()).await.unwrap();
    let error = session
        .generate(GenerateRequest { ir: json!({}), target: "x".into(), options: Default::default() })
        .await
        .unwrap_err();
    assert!(matches!(error, CallError::Failed(_)), "{error:?}");
    assert_eq!(log.methods().last().map(String::as_str), Some(methods::EXIT));
}

Replace the traversal path with the daemon's exact rejected form (read validate_generate_result and its daemon test compatibility_invoke_rejects_unsafe_generated_artifacts) and assert the exact error text.

Move the daemon's validation unit tests (session/validation.rs tests for workspace, generate and compile results) with the code.

• Step 2: Run the tests and confirm they fail

Run: cargo test -p morphir-host-native --test checked_connection
Expected: compile error no CheckedConnection in the root.

• Step 3: Implement and move

Move validate_method_result, validate_method_result_async (renamed validate_result), and every helper they use into results.rs. The daemon controller and SpawnedProcessSession call morphir_host_native::validate_result(...).await.map_err(DaemonError::from). Implement CheckedConnection and activate. Export from lib.rs: pub use results::{CheckedConnection, validate_result}; pub use activate::{ActivatedGuest, activate}; pub use native::NativeChannel; and the extism and process modules.

• Step 4: Run all suites

Run: cargo test -p morphir-host-native && cargo test -p morphir-host && cargo test -p morphir-daemon --lib --tests
Run: the ignored process suites from task 5 and mise run test:extism.
Run: cargo clippy --all-targets --all-features -p morphir-host -p morphir-host-native -p morphir-daemon -- -D warnings && cargo fmt --all -- --check && cargo check -p morphir-host --target wasm32-unknown-unknown && cargo doc -p morphir-host -p morphir-host-native --no-deps
Add a CHANGELOG [Unreleased] entry: Added morphir-host-native; Changed: morphir-host gains Channel::abort, ExpectedChecks and describe; the daemon's process, Extism and native transports run on morphir-host-native.

• Step 5: Commit, then whole-branch review, then PR

git add crates CHANGELOG.md Cargo.lock
git commit -m "feat(host-native): validate method results in a checked connection and activate installed guests"

The PR opens after the whole-branch review. It refers to finos/morphir-rust#237 and #240.

[DamianReeves]

Task 8: CLI probe and publish describe through morphir_host::describe

Status: Done in #989 (merged as d79afa3).

This task and task 9 run in finos/morphir after the task 7 PR merges and main CI passes on the merge commit.

Files:

• Modify: crates/morphir/Cargo.toml (add morphir-host-native), ecosystem/morphir-rust (pin)
• Modify: crates/morphir/src/commands/extension/probe.rs, crates/morphir/src/commands/extension.rs (describe_publish_artifact)
• Modify: crates/morphir/tests/support/process_bundle.rs

Interfaces:

• Consumes: morphir_host::{describe, Description, DescriptionSource}, morphir_host_native::process::{ProcessChannel, ProcessLaunch}.

• The probe keeps its outputs: ProbeSource::Describe or ProbeSource::SessionFallback, the same check_claims / check_session calls, and the same DistributionError::Probe(text) mapping.

• Step 1: Change the three call sites

In each of probe.rs::claims, describe_publish_artifact, and tests/support/process_bundle.rs::from_executable, replace:

let transport = SpawnedProcessTransport::spawn(launch).await.map_err(...)?;
let description = transport.describe(<params>).await.map_err(...)?;

with:

let channel = morphir_host_native::process::ProcessChannel::spawn(launch)
    .await
    .map_err(|error| <same mapping as spawn today>)?;
let description = morphir_host::describe(channel, &host_config(), &<the launch id>)
    .await
    .map_err(|error| <same mapping as describe today>)?;

ProcessLaunch now comes from morphir_host_native::process. Match on morphir_host::DescriptionSource in place of the daemon's type. In tests/support/process_bundle.rs, use crate::...::host_config() if the support module can reach it; otherwise build morphir_host::HostConfig::new(PeerInfo { kind: PeerKind::Cli, name: "morphir-cli".into(), version: env!("CARGO_PKG_VERSION").into() }) there, so the test support stops hardcoding MEP_VERSION.

• Step 2: Run the probe and publish tests

Run: cargo test --locked --package morphir --test cli_integration extension_probe -- --test-threads=1 and cargo test --locked --package morphir --test cli_integration process_bundle -- --test-threads=1
Expected: every test named in tests/cli_integration/extension_probe.rs and tests/support/process_publish.rs passes.

• Step 3: Commit

git add crates/morphir ecosystem/morphir-rust Cargo.lock
git commit -m "feat(cli): probe and describe extensions through morphir-host"

[DamianReeves]

Task 9: CLI one-shot calls through morphir_host::Session

Status: Done in #989 (merged as d79afa3).

Files:

• Create: crates/morphir/src/extensions/guest.rs
• Modify: crates/morphir/src/extensions.rs (probe_process, invoke_process, invoke_frontend, invoke_backend, invoke_workspace_discovery; delete invoke_installed, installed_loaded, invoke_loaded, session_failure, failed_session_message)
• Modify: crates/morphir/src/commands/ui/provider/extension.rs (invoke_installed)
• Keep: open_frontend_session, open_backend_session, open_loaded (workbench; part 3)

Interfaces:

• Produces in crate::extensions::guest (crate-private):

/// A checked MEP connection to one provider, ready to open.
pub(crate) type ProviderConnection =
    morphir_host_native::CheckedConnection<morphir_host::JsonRpcConnection<Box<dyn morphir_host::Channel>, morphir_host::ExpectedChecks>>;

/// Open a connection to a configured process provider.
pub(crate) async fn configured(launch: ProcessLaunch, provider: &str) -> Result<ProviderConnection, CliError>;

/// Open a connection to a resolved provider that speaks MEP
/// (`NativeMep`, `ProcessMep` or `WasmMep`).
pub(crate) async fn resolved(
    home: &MorphirHome,
    workspace: &Path,
    provider: &str,
    mode: InvocationMode,
    native: Option<&NativeExtension>,
    installed: Option<&InstalledExtensionSnapshot>,
) -> Result<ProviderConnection, CliError>;

/// Open, make one call, close. Errors keep the CLI's texts.
pub(crate) async fn call_once<P: Serialize, R: DeserializeOwned>(
    connection: ProviderConnection,
    provider: &str,
    method: &str,
    request: P,
) -> Result<R, CliError>;

/// Open and read what the provider negotiated, then close.
pub(crate) async fn negotiate(connection: ProviderConnection, provider: &str) -> Result<NegotiatedProvider, CliError>;

• Error texts (unchanged from today):

  • spawn: Failed to start provider '{provider}': {error}
  • handshake failure: Provider '{provider}' failed during initialize: {error}
  • rejected call: Provider '{provider}' rejected '{method}': {error}, plus ; orderly shutdown also failed: {close} when the orderly close fails
  • failed call: Provider '{provider}' failed during {method}: {error}
  • shutdown failure after success: Provider '{provider}' failed during shutdown: {error}
  • installed verification: Failed to verify installed provider '{provider}': {error}; activation: Failed to activate installed provider '{provider}': {error}

• Session::call returns CallError::Rejected or CallError::Failed; map Rejected to the rejected text and Failed to the failed text. The old "; transport state is indeterminate" suffix appears when the host error is HostError::Channel { state: Indeterminate, .. }.

• Step 1: Write a failing test that pins the rejected-call text

In crates/morphir/src/extensions.rs tests, add a test that compiles through a configured process provider script that answers morphir.frontend.compile with an RPC error, and asserts the exact text Provider 'fixture' rejected 'morphir.frontend.compile': RPC error -32001: does not compile. Reuse the unix script pattern already used by gleam_legacy_selector_preserves_an_installed_provider_with_that_id or the extension_probe.rs fixture guest; mark it #[cfg(unix)].

Run it on the current code first. It must PASS before the change (it pins today's text). Record that run.

• Step 2: Implement guest.rs and rewire the call sites

• probe_process(launch, provider): guest::negotiate(guest::configured(launch, provider).await?, provider).

• invoke_process(launch, provider, method, request): guest::call_once(guest::configured(launch, provider).await?, provider, method, request).

• invoke_frontend, invoke_backend, invoke_workspace_discovery: keep the NativeDirect arm. Replace the NativeMep and ProcessMep | WasmMep arms with guest::call_once(guest::resolved(home, workspace, id, mode, resolved.native_extension(), resolved.installed_snapshot()).await?, id, method, request).

• guest::resolved for NativeMep uses NativeChannel::new(native); for installed modes it calls activate_installed_snapshot(home, snapshot) then morphir_host_native::activate(artifact, workspace).

• ui/provider/extension.rs::invoke_installed: open guest::resolved(...) for the snapshot, Session::open, check the negotiated workspace capability with the same test as today, call methods::WORKSPACE_DISCOVER, close. Keep its error texts.

• Delete the typestate helpers listed above and the morphir_daemon::extensions imports they needed.

• Step 3: Run the CLI gates and the compatibility suites

Run: cargo fmt --package morphir --package integration-tests --check && cargo clippy --locked --package morphir --package integration-tests --all-targets -- -D warnings && cargo test --locked --package morphir -- --test-threads=4
Expected: all pass, including the new text-pinning test and generate_rejects_traversal_from_an_installed_provider_without_writing.
Run: mise run ci:fetch-published-bundles && cargo build --locked --release --package morphir && mise run test:host-compatibility
Expected: avro, openapi, python and rust pass locally. elm_reference and scala_elm need Linux x86_64 or MORPHIR_ELM_EXTENSION_BIN / MORPHIR_SCALA_ELM_EXTENSION_BIN; PR CI runs them.

• Step 4: Commit and open the PR

git add crates/morphir Cargo.lock
git commit -m "feat(cli): run one-shot extension calls through morphir-host sessions"

The PR refers to #963 and #959. The workbench pool and the registry remain for part 3.

[DamianReeves]

After part 2

Start the part 3 plan from these facts:

• The CLI's only remaining daemon session users are the workbench (open_frontend_session, open_backend_session, open_loaded, sessions.rs, SessionHandle) and the registry (ExtensionRegistry, ResolvedFrontend, ResolvedBackend, InvocationMode, InvocationPolicy).
• Pool<K> in morphir-host replaces SessionReuseInvoker, with its 8 tests ported.
• Registry and GuestSource in morphir-host (portable), with the native, process and installed sources in morphir-host-native.
• The daemon's typestate Session, the kameo actor, ExtensionResponseExt, and the daemon copies of the transports can then be removed.

[DamianReeves]

Part 2 shipped

• feat(host-native): native channels, the describe probe and negotiation rules on the host crates morphir-rust#263 (merged as 3b4e4fe) contains:
  • morphir-host-native: ProcessChannel, NativeChannel, ExtismChannel with the moved Extism container, validate_result, CheckedConnection and activate.
  • morphir-host gains Channel::abort, ExpectedChecks and the moved negotiation rules, describe/describe_with (the moved install probe) and HostError::Io.
  • The daemon delegates to both crates, and its public API does not change.
• feat(cli): run extension probes and one-shot calls through morphir-host sessions #989 (merged as d79afa3) contains:
  • The CLI install probe and publish describe run through morphir_host::describe.
  • One-shot calls for configured, installed and native MEP providers, and the workbench's installed workspace provider, run through morphir_host::Session and CheckedConnection.
  • The host compatibility suite passed on Linux, including the new Elm 0.4.0 describe path.

Changes to the plan during execution

Ruling	What changed	Cost if wrong
R3	The ProcessLaunch read accessor is configured_request_timeout(), because request_timeout() is the builder.	A rename.
R4	HostError::Io shows IO error: {0} and maps to DaemonError::Io, so the daemon text is kept.	One extra public variant.
Task 5	The describe fallback keeps the full launch expectation through the hidden describe_with.	None.
R5	Final-review fixes: the boxed channel forwards abort, the process test timeouts are stable, the public API is tidied, and the CHANGELOG is corrected.	A hidden helper someone wanted must be re-exported.
PR review	activate has a doc example. One timeout now covers the whole process request.	None.
R6	CLI error texts keep the Extension error: prefix through DaemonError::from.	None.
R8	Accepted text change: transport failures that start from an I/O or JSON decode error now read Extension error: IO error: .... The CHANGELOG records this.	A script that matches the old text breaks until part 3.
R9	GuestSource in the CLI's guest.rs, plus a CHANGELOG entry.	None.

Carried into part 3

• ChannelError carries the root HostError kind, which undoes the R8 text change.
• builtin_provider in the CLI rebuilds the built-in for NativeMep. The part 3 Registry exposes the native provider for MEP mode.
• Missing tests: the UI installed workspace provider, and the rejected-call text for installed providers.
• The six near-identical match arms in extensions.rs, and the duplicated failure formatting (session_failure against guest::failure, and also_failed_to_shut_down).
• The workbench pool (sessions.rs, SessionHandle), the provider registry, and the removal of the daemon's typestate session, actor and ExtensionResponseExt.

[DamianReeves]

This plan is complete. Further tracking of the host library work is in beads (morphir-opua.1 and its children); the GitHub issues under #958 link to the beads.