From 341c4c59b8a08a9aad187afcd328816d949136ae Mon Sep 17 00:00:00 2001 From: Sam Lijin Date: Thu, 12 Nov 2026 15:20:02 -0800 Subject: [PATCH 1/3] feat(wrapper): record canonical commands, errors, and child exit status --- TELEMETRY.md | 6 + .../baml_cli/tests/embedded_telemetry_e2e.rs | 484 ++++++++++++++++-- .../tests/wrapper_command_context_e2e.rs | 367 +++++++++++++ baml_language/wrapper/README.md | 27 +- baml_language/wrapper/baml_src/entry.baml | 29 +- .../wrapper/baml_src/invocation.baml | 12 +- baml_language/wrapper/baml_src/selection.baml | 6 +- baml_language/wrapper/baml_src/selectors.baml | 8 +- 8 files changed, 872 insertions(+), 67 deletions(-) create mode 100644 baml_language/crates/baml_cli/tests/wrapper_command_context_e2e.rs diff --git a/TELEMETRY.md b/TELEMETRY.md index 788041a8920..f2961326f4d 100644 --- a/TELEMETRY.md +++ b/TELEMETRY.md @@ -80,6 +80,8 @@ You may also opt out by setting the cross-tool environment variable The Rust and BAML toolchain wrappers share an independent PostHog event schema. Each invocation emits one canonical command event, such as `baml`, `baml check`, `baml toolchain status`, `baml toolchain update`, or `baml self-update`, with `event_kind = "invocation"`. All flags and argument values are discarded: `baml --help` becomes `baml`, and `baml toolchain update --help` becomes `baml toolchain update`. Only recognized command names enter telemetry; unknown commands collapse to the nearest recognized parent. Delegated CLI commands use their recognized top-level name, without inspecting the CLI's nested arguments. Internal telemetry and replacement helpers are excluded. +The canonical `baml` label does not prove empty argv: it also covers root help, version, unknown commands, and `--` termination. + Every wrapper event includes `schema_version = 2`, its canonical `command`, and a `client` discriminator: `rust-wrapper` or `baml-in-baml`. Both implementations also emit `wrapper_resolution`, `wrapper_install`, `wrapper_selection`, `wrapper_update`, and `wrapper_self_update` for bounded decisions and outcomes. `wrapper_unhandled_error` records errors escaping the wrapper command handler, including argument parsing and failed handoffs. Recovered errors and delegated CLI nonzero exit codes do not count as unhandled wrapper errors. A delegated CLI keeps its separate telemetry. Events include coarse OS/architecture/CPU/CI information, a random anonymous identity and session, and a salted project-root hash. Source categories, toolchain kinds, and outcomes are bounded labels; argument values, selectors, raw paths, panic payloads, and error text are not sent. @@ -92,6 +94,10 @@ Wrapper events are written locally before a toolchain handoff. If a handoff fail The Rust wrapper sends only PostHog usage events. Release builds of the BAML-native wrapper additionally enable BCS runtime recording through an embedded public ingestion credential. BCS uses the existing packed-runtime policy: delivery runs on a background worker and shutdown waits for its normal bounded drain. The wrapper selects the existing `on_initial_telemetry_failure = "ignore"` policy. The PostHog queue retains its separate retry behavior. Set `BAML_TELEMETRY=off` to disable the BAML wrapper’s BCS recording. +The BAML-native wrapper records error payloads at its explicit dispatch, argument-parser, management-parser/preflight/execution, toolchain-selection/acquisition, passive freshness/refresh and self-update boundaries. It does not request full input snapshots or argument-bearing returned objects. Delegated and version calls retain only child exit code and signal in a `baml.sys.ProcessExit`; child stdout/stderr remain inherited and uncaptured. A delegated CLI/project keeps its own recording policy. Functions unwound by `sys.exit` have no returned output. Error payloads are retained as supplied and can quote offending arguments, selectors or paths, including secrets; suppressing input snapshots does not redact errors. Internal telemetry/replacement helpers bypass argument parsing, and command help bypasses management parsing and self-update. Capture applies to the requested invocation, not all descendants; errors recovered entirely inside an untraced utility retain its automatic policy. Other automatic functions do not acquire value capture from these requests or from `high`. For the packed wrapper, `BAML_TELEMETRY=on` keeps the baked default level (normally medium), while `off` disables BCS recording. HTTP network spans still exclude bodies. See the [wrapper capture boundary table](baml_language/wrapper/README.md) for exact scopes. This BCS policy does not change PostHog's metadata-only contract, identities, destinations or existing launch metadata. + +The BAML-native wrapper also attaches the same bounded canonical `command` to the foreground `user.dispatch` span and descendants, independently of PostHog and function value capture. Helpers remove this key in their dispatch scope. This does not update process launch metadata or add an identity; join dispatch to `processes` by `process_id` for process outcomes. See the [wrapper query example](baml_language/wrapper/README.md) for counting invocations without duplicating inherited spans. + ## BCS environment metadata Enabled cloud runtime recordings automatically include the following scalar properties under the system-populated `baml.ctx.` prefix in the recorded process context's `metadata` map. The recording header's existing `initial_context_cas_id` references this context, which travels through the existing recording/CAS upload pipeline. This does not add a network request or change the recording format. The properties describe the process running BAML, including SDK calls, CLI commands and packed executables that select BCS delivery. diff --git a/baml_language/crates/baml_cli/tests/embedded_telemetry_e2e.rs b/baml_language/crates/baml_cli/tests/embedded_telemetry_e2e.rs index ca11ebddca2..0e796a76f85 100644 --- a/baml_language/crates/baml_cli/tests/embedded_telemetry_e2e.rs +++ b/baml_language/crates/baml_cli/tests/embedded_telemetry_e2e.rs @@ -789,7 +789,7 @@ async fn wrapper_build_embeds_named_api_key_for_both_release_variants() { } #[test] -fn wrapper_command_boundaries_record_spans_without_values() { +fn wrapper_boundaries_capture_errors_and_exit_status_without_inputs() { let built = common::ensure_built(); let repo = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../.."); let temp = tempfile::tempdir().unwrap(); @@ -800,7 +800,31 @@ fn wrapper_command_boundaries_record_spans_without_values() { } else { "wrapper" }); - let output = Command::new(if cfg!(windows) { "python" } else { "python3" }) + // Explicit local delivery and an isolated home keep every negative fixture + // independent of saved credentials, embedded release keys and ambient context. + let local = |program: &Path| { + let mut command = Command::new(program); + command.env_clear(); + // Windows needs SystemRoot for side-by-side assemblies; preserve only + // OS process-launch settings, never ambient BAML or Boundary settings. + for key in ["PATH", "SystemRoot", "WINDIR", "PATHEXT", "TEMP", "TMP"] { + if let Some(value) = std::env::var_os(key) { + command.env(key, value); + } + } + command + .env("HOME", &home) + .env("USERPROFILE", &home) + .env("BAML_HOME", home.join(".baml")) + .env("BOUNDARY_API_KEY", "local") + .env("BAML_CLI_ALLOW_DIRECT", "1") + .env("DEV_BAML_CLI_DISABLE_AGENT_DETECTION", "1") + .env("BAML_TELEMETRY", "off") + .env("DO_NOT_TRACK", "1") + .current_dir(temp.path()); + command + }; + let output = local(Path::new(if cfg!(windows) { "python" } else { "python3" })) .arg(repo.join("baml_language/wrapper/build")) .arg("--cli") .arg(&built.baml_cli) @@ -808,89 +832,443 @@ fn wrapper_command_boundaries_record_spans_without_values() { .arg(&built.baml_pack_host) .arg("--output") .arg(&binary) - .env("BAML_HOME", home.join(".baml")) - .env("BAML_TELEMETRY", "off") - .env("DO_NOT_TRACK", "1") .output() .unwrap(); succeeded(&output); - for (scenario, args, success) in [ - ("help", vec!["--help"], true), - ("update", vec!["toolchain", "update"], false), - ("self-update", vec!["self-update"], false), - ("parse-error", vec!["--project"], false), - ("helper", vec!["__flush-wrapper-telemetry"], true), + // A portable child fixture returns a known nonzero result without treating + // it as a thrown wrapper error, and leaves stdout/stderr inherited. + let child_project = temp.path().join("child"); + std::fs::create_dir(&child_project).unwrap(); + common::write_project( + &child_project, + r#" +function main() -> void { + baml.io.println("delegated-output-sentinel"); + baml.sys.exit(if (baml.sys.argv().includes("exit-zero")) { 0 } else { 7 }) +} +"#, + ); + let child = temp.path().join(if cfg!(windows) { + "child.exe" + } else { + "child-bin" + }); + succeeded( + &local(&built.baml_cli) + .args(["pack", "main", "--from"]) + .arg(&child_project) + .arg("--output") + .arg(&child) + .output() + .unwrap(), + ); + // A cached channel installation lets passive failures recover without any + // downloads, while the delegated command still completes successfully. + let managed = home.join(".baml/toolchains/0.0.79"); + std::fs::create_dir_all(managed.join("bin")).unwrap(); + std::fs::copy( + &child, + managed.join("bin").join(if cfg!(windows) { + "baml-cli.exe" + } else { + "baml-cli" + }), + ) + .unwrap(); + std::fs::write(managed.join("VERSION"), "0.0.79\n").unwrap(); + std::fs::write( + home.join(".baml/state.toml"), + "[channels.canary]\nactive_version = \"0.0.79\"\n", + ) + .unwrap(); + let missing = temp + .path() + .join("missing-project-sentinel") + .display() + .to_string(); + let sentinel_project = temp.path().join("project-sentinel"); + std::fs::create_dir(&sentinel_project).unwrap(); + let sentinel_project = sentinel_project.display().to_string(); + for (scenario, args, code) in [ + ("help", vec!["--project", &sentinel_project, "--help"], 0), + ("parse-error", vec!["--project", "--sentinel"], 1), + ("project-error", vec!["--project", &missing, "check"], 1), + ( + "selection-error", + vec!["check", "selection-input-sentinel"], + 1, + ), + ( + "command-error", + vec!["toolchain", "status", "--sentinel-option"], + 1, + ), + ("preflight-error", vec!["toolchain", "update"], 1), + ("execute-error", vec!["toolchain", "uninstall", "0.0.77"], 1), + ("list", vec!["toolchain", "list"], 0), + ("self-update", vec!["self-update"], 1), + ("helper", vec!["__flush-wrapper-telemetry"], 0), + ("delegated", vec!["check", "delegated-input-sentinel"], 7), + ("version", vec!["--version"], 7), + ("passive-errors", vec!["check", "exit-zero"], 0), + ("high", vec!["--help"], 0), + ("on", vec!["--help"], 0), + ("off", vec!["--help"], 0), ] { - let output = Command::new(&binary) + let toolchain = match scenario { + "delegated" | "version" => child.to_str().unwrap(), + "list" => "0.0.78", + "passive-errors" => "canary", + _ => "invalid-version-sentinel", + }; + let level = match scenario { + "high" | "on" | "off" => scenario, + _ => "medium", + }; + if scenario == "passive-errors" { + std::fs::write(home.join(".baml/config.toml"), "bad-config-sentinel = [").unwrap(); + } + let output = local(&binary) .args(args) - .current_dir(temp.path()) - .env("BAML_HOME", home.join(".baml")) - .env("BAML_TELEMETRY", "medium") + .env("BAML_TELEMETRY", level) .env( "BAML_CONTEXT", json!({"metadata": {"scenario": scenario}}).to_string(), ) - .env("BAML_TOOLCHAIN", "invalid-version") + .env("BAML_TOOLCHAIN", toolchain) .env("BAML_MANIFEST_BASE_URL", "invalid://release-source") - .env_remove("BOUNDARY_API_KEY") - .env("DO_NOT_TRACK", "1") .output() .unwrap(); - assert_eq!(output.status.success(), success, "{scenario}: {output:?}"); + assert_eq!(output.status.code(), Some(code), "{scenario}: {output:?}"); + if scenario == "passive-errors" { + std::fs::remove_file(home.join(".baml/config.toml")).unwrap(); + } + if matches!(scenario, "delegated" | "version" | "passive-errors") { + assert!(String::from_utf8_lossy(&output.stdout).contains("delegated-output-sentinel")); + } + } + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + let invalid_binary = temp.path().join("launch-failure-sentinel"); + // A missing shebang interpreter guarantees a spawn error; a plain + // invalid executable may be launched through the platform shell. + std::fs::write( + &invalid_binary, + format!("#!{}\n", temp.path().join("missing-interpreter").display()), + ) + .unwrap(); + std::fs::set_permissions(&invalid_binary, std::fs::Permissions::from_mode(0o755)).unwrap(); + let output = local(&binary) + .args(["check", "launch-input-sentinel"]) + .env("BAML_TOOLCHAIN", &invalid_binary) + .env("BAML_TELEMETRY", "medium") + .env( + "BAML_CONTEXT", + json!({"metadata": {"scenario": "launch-error"}}).to_string(), + ) + .output() + .unwrap(); + assert_eq!(output.status.code(), Some(1), "{output:?}"); + } + #[cfg(unix)] + { + // An unconnected datagram stdout fails with EDESTADDRREQ, keeping stderr + // available for diagnostics. Avoid BrokenPipe: its separate structured + // error handling must not affect this panic-capture assertion. + let writer = std::os::unix::net::UnixDatagram::unbound().unwrap(); + let output = local(&binary) + .arg("--help") + .env("BAML_TELEMETRY", "medium") + .env( + "BAML_CONTEXT", + json!({"metadata": {"scenario": "panic"}}).to_string(), + ) + .stdout(std::process::Stdio::from(std::os::fd::OwnedFd::from( + writer, + ))) + .output() + .unwrap(); + assert_eq!(output.status.code(), Some(1), "{output:?}"); } - let query = |sql: &str| { - let output = Command::new(&built.baml_cli) + let output = local(&built.baml_cli) .args(["query", "--local", "--format", "json", "--from"]) .arg(&home) .arg(sql) - .env("BAML_HOME", home.join(".baml")) - .env("BAML_CLI_ALLOW_DIRECT", "1") - .env("BAML_TELEMETRY", "off") - .env("DO_NOT_TRACK", "1") .output() .unwrap(); succeeded(&output); serde_json::from_slice::(&output.stdout).unwrap()["rows"].take() }; + let spans = query("SELECT context_metadata['scenario'], span_name, status, input_args, output_value, error_value, + baml_value_state(input_args), baml_value_state(output_value), baml_value_state(error_value), + context_metadata['command'] FROM spans WHERE span_type = 'function'"); + let span = |scenario: &str, name: &str| { + let rows: Vec<_> = spans + .as_array() + .unwrap() + .iter() + .filter(|row| row[0] == scenario && row[1] == name) + .collect(); + assert_eq!(rows.len(), 1, "{scenario}/{name}: {rows:?}"); + Value::Array(rows[0].as_array().unwrap()[2..].to_vec()) + }; + let parsed = span("help", "user.WrapperInvocation.parse"); + assert_eq!(parsed[0], "return"); + assert_eq!(parsed[4], "no_value"); + assert_eq!( + parsed[5], "no_value", + "parser results must not reintroduce argv or project paths" + ); + assert_eq!(parsed[7], "baml"); + let returned = span("help", "user.dispatch"); + assert_eq!(returned[0], "return"); + assert_eq!( + returned[5], "no_value", + "null returns need no value snapshot" + ); + for (scenario, name, class, message) in [ + ( + "parse-error", + "user.WrapperInvocation.parse", + "baml.errors.InvalidArgument", + "--project requires a path", + ), + ( + "project-error", + "user.WrapperInvocation.parse", + "baml.errors.InvalidArgument", + "missing-project-sentinel", + ), + ( + "selection-error", + "user.WrapperInvocation.selection_impl", + "user.InvalidToolchainVersion", + "invalid-version-sentinel", + ), + ( + "command-error", + "user.management_command", + "user.UnexpectedManagementArgument", + "", + ), + ( + "preflight-error", + "user.<(user.UpdateCommand as user.ManagementCommand)>.preflight", + "user.InvalidToolchainVersion", + "invalid-version-sentinel", + ), + ( + "execute-error", + "user.<(user.UninstallCommand as user.ManagementCommand)>.execute", + "baml.errors.InvalidArgument", + "toolchain 0.0.77 is not installed", + ), + ( + "self-update", + "user.WrapperUpdater.update", + "baml.errors.InvalidArgument", + "HTTPS", + ), + ] { + let failed = span(scenario, name); + assert_eq!(failed[0], "user_error", "{scenario}: {failed}"); + assert_eq!( + failed[3]["$class"], "baml.errors.Context", + "{scenario}: {failed}" + ); + assert_eq!(failed[3]["error"]["$class"], class, "{scenario}: {failed}"); + assert!( + failed[3]["error"].to_string().contains(message), + "{scenario}: {failed}" + ); + assert_eq!(failed[6], "present"); + assert_eq!(span(scenario, "user.dispatch")[0], "user_error"); + } + let command_error = span("command-error", "user.management_command"); + assert_eq!(command_error[4], "no_value"); + assert_eq!( + command_error[3]["error"]["argument"], "--sentinel-option", + "full errors can quote an argument even though argv is not captured" + ); + assert_eq!(command_error[7], "baml toolchain status"); + let command = span("execute-error", "user.management_command"); + assert_eq!( + command[5], "no_value", + "typed command options can contain input values" + ); + let executed = span( + "list", + "user.<(user.ListCommand as user.ManagementCommand)>.execute", + ); + assert_eq!(executed[0], "return"); + assert_eq!(executed[5], "no_value"); + assert_eq!( + span("list", "user.dispatch")[0], + "return", + "recovered preflight failures stay recovered" + ); + let acquired = span( + "list", + "user.<(user.VersionSelector as user.ToolchainSelector)>.ensure_installed", + ); + assert_eq!( + acquired[0], "user_error", + "best-effort acquisition errors remain visible before recovery" + ); + assert_eq!( + acquired[3]["error"]["$class"], + "baml.errors.InvalidArgument" + ); + assert!( + acquired[3]["error"]["message"] + .as_str() + .unwrap() + .contains("HTTPS") + ); + assert_eq!(acquired[4], "no_value"); + assert_eq!(acquired[5], "no_value"); + assert_eq!(acquired[7], "baml toolchain list"); + assert_eq!( + span( + "list", + "user.<(user.ListCommand as user.ManagementCommand)>.preflight" + )[0], + "return" + ); + let selected = span("delegated", "user.WrapperInvocation.selection_impl"); + assert_eq!(selected[0], "return"); + assert_eq!( + selected[5], "no_value", + "selection results can contain selectors and paths" + ); + for (scenario, command, code) in [ + ("delegated", "baml check", 7), + ("version", "baml", 7), + ("passive-errors", "baml check", 0), + ] { + let child_result = span(scenario, "user.run_toolchain"); + assert_eq!(child_result[0], "return"); + assert_eq!(child_result[4], "no_value"); + assert_eq!(child_result[2]["$class"], "baml.sys.ProcessExit"); + assert_eq!(child_result[2]["exit_code"], code); + assert_eq!(child_result[2]["signal"], Value::Null); + // Query results also carry $class/$definition type metadata. Only the + // data fields may hold returned values, never argv or stdio buffers. + let mut data_fields = child_result[2] + .as_object() + .unwrap() + .keys() + .filter(|key| !key.starts_with('$')) + .map(String::as_str) + .collect::>(); + data_fields.sort_unstable(); + assert_eq!(data_fields, ["exit_code", "signal"]); + assert_eq!(child_result[7], command); + } + let refreshed = span( + "passive-errors", + "user.ChannelSelector.refresh_channel_impl", + ); + assert_eq!(refreshed[0], "user_error"); + assert_eq!( + refreshed[3]["error"]["$class"], + "baml.errors.InvalidArgument" + ); + assert!( + refreshed[3]["error"]["message"] + .as_str() + .unwrap() + .contains("config.toml") + ); + assert_eq!(refreshed[7], "baml check"); + assert_eq!(query("SELECT COUNT(*) FROM spans WHERE span_name = 'user.ChannelSelector.warn_if_outdated_impl' + AND context_metadata['scenario'] = 'passive-errors' AND status = 'user_error' + AND baml_value_state(error_value) = 'present'"), json!([[2]])); + assert_eq!( + span("passive-errors", "user.dispatch")[0], + "cancel_error", + "recovered passive errors do not change the successful child exit" + ); assert_eq!( query( - "SELECT context_metadata['scenario'], span_name, status FROM spans - WHERE span_type = 'function' AND span_name IN ( - 'user.WrapperInvocation.parse', 'user.WrapperUpdater.update', - 'user.<(user.UpdateCommand as user.ManagementCommand)>.preflight', 'user.dispatch') - ORDER BY context_metadata['scenario'], span_name" + "SELECT DISTINCT status FROM processes WHERE context_metadata['scenario'] = 'passive-errors'" ), - json!([ - ["help", "user.WrapperInvocation.parse", "return"], - ["help", "user.dispatch", "return"], - ["helper", "user.dispatch", "return"], - ["parse-error", "user.WrapperInvocation.parse", "user_error"], - ["parse-error", "user.dispatch", "user_error"], - ["self-update", "user.WrapperInvocation.parse", "return"], - ["self-update", "user.WrapperUpdater.update", "user_error"], - ["self-update", "user.dispatch", "user_error"], - [ - "update", - "user.<(user.UpdateCommand as user.ManagementCommand)>.preflight", - "user_error" - ], - ["update", "user.WrapperInvocation.parse", "return"], - ["update", "user.dispatch", "user_error"], - ]) + json!([["success"]]) + ); + let exited = span("delegated", "user.dispatch"); + assert_eq!(exited[0], "cancel_error"); + assert_eq!(exited[5], "no_value", "sys.exit does not return an output"); + assert_eq!(exited[6], "present"); + assert_eq!(exited[3]["error"]["$class"], "baml.panics.Exit"); + assert_eq!(exited[3]["error"]["code"], 7); + // Verify the persisted recording, not just the source flags: no function + // receives an input snapshot, and only the child status has a result value. + assert_eq!( + query( + "SELECT COUNT(*) FROM spans WHERE span_type = 'function' + AND baml_value_state(input_args) != 'no_value'" + ), + json!([[0]]) + ); + assert_eq!( + query( + "SELECT COUNT(*) FROM spans WHERE span_type = 'function' + AND span_name != 'user.run_toolchain' AND baml_value_state(output_value) != 'no_value'" + ), + json!([[0]]) + ); + assert_eq!( + query("SELECT COUNT(*) FROM processes WHERE context_metadata['scenario'] = 'off'"), + json!([[0]]) ); assert_eq!( query( - "SELECT COUNT(*) FROM spans WHERE span_type = 'function' AND span_name IN ( - 'user.WrapperInvocation.parse', 'user.WrapperUpdater.update', - 'user.<(user.UpdateCommand as user.ManagementCommand)>.preflight', 'user.dispatch') - AND (baml_value_state(input_args) = 'present' - OR baml_value_state(output_value) = 'present' - OR baml_value_state(error_value) = 'present')" + "SELECT COUNT(*) FROM spans WHERE context_metadata['scenario'] = 'helper' + AND span_name = 'user.WrapperInvocation.parse'" ), json!([[0]]) ); + let on = span("on", "user.WrapperInvocation.parse"); + assert_eq!(on[4], "no_value"); + assert_eq!(on[5], "no_value"); + assert_eq!( + on[7], "baml", + "on keeps the packed default and canonical context without inputs" + ); + #[cfg(unix)] + { + let panicked = span("panic", "user.dispatch"); + assert_eq!(panicked[0], "panic_error"); + assert_eq!( + panicked[3]["error"]["$class"], + "baml.panics.HostUnavailable" + ); + assert_eq!(panicked[6], "present"); + assert_eq!(panicked[5], "no_value"); + let launch = span("launch-error", "user.run_toolchain"); + assert_eq!(launch[0], "user_error"); + assert_eq!(launch[3]["error"]["$class"], "baml.errors.Io"); + assert!( + launch[3]["error"]["message"] + .as_str() + .unwrap() + .contains("launch-failure-sentinel") + ); + assert_eq!(launch[4], "no_value"); + assert_eq!(launch[5], "no_value"); + assert_eq!(launch[7], "baml check"); + } + let automatic = span("high", "user.root_help"); + assert_eq!(automatic[0], "return"); + assert_eq!(automatic[4], "no_value"); + assert_eq!( + automatic[5], "no_value", + "high promotes spans without enabling value capture" + ); + assert_eq!(automatic[6], "no_value"); } #[tokio::test(flavor = "multi_thread", worker_threads = 2)] diff --git a/baml_language/crates/baml_cli/tests/wrapper_command_context_e2e.rs b/baml_language/crates/baml_cli/tests/wrapper_command_context_e2e.rs new file mode 100644 index 00000000000..abcf6b675f1 --- /dev/null +++ b/baml_language/crates/baml_cli/tests/wrapper_command_context_e2e.rs @@ -0,0 +1,367 @@ +//! The packed wrapper's actual local recordings, queried in a fresh process. +//! BAML cannot inspect the persisted process outcome or span announcements itself. +mod common; + +use std::{ + path::Path, + process::{Command, Output}, +}; + +use serde_json::{Value, json}; + +fn succeeded(output: &Output) { + assert!(output.status.success(), "{output:?}"); +} + +fn local_command(binary: &Path, home: &Path) -> Command { + let mut command = Command::new(binary); + command + .env("BAML_HOME", home) + .env("BOUNDARY_API_KEY", "local") + .env_remove("BOUNDARY_PROJECT") + .env_remove("BOUNDARY_API_URL") + .env_remove("BAML_CONTEXT") + .env("BAML_TELEMETRY", "off") + .env("DO_NOT_TRACK", "1") + .env("BAML_CLI_ALLOW_DIRECT", "1") + .env("DEV_BAML_CLI_DISABLE_AGENT_DETECTION", "1"); + command +} + +#[test] +fn canonical_commands_join_local_process_outcomes_with_posthog_disabled() { + let built = common::ensure_built(); + let repo = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../.."); + let temp = tempfile::tempdir().unwrap(); + let project = temp.path().join("project"); + std::fs::create_dir_all(&project).unwrap(); + common::write_project(&project, "function main() -> int { 7 }\n"); + let home = temp.path().join("home/.baml"); + let binary = temp.path().join(if cfg!(windows) { + "wrapper.exe" + } else { + "wrapper" + }); + let python = Path::new(if cfg!(windows) { "python" } else { "python3" }); + succeeded( + &local_command(python, &home) + .arg(repo.join("baml_language/wrapper/build")) + .arg("--cli") + .arg(&built.baml_cli) + .arg("--pack-host") + .arg(&built.baml_pack_host) + .arg("--output") + .arg(&binary) + .output() + .unwrap(), + ); + let project_path = project.to_str().unwrap(); + // The launch command intentionally conflicts with our computed label: the + // wrapper owns this key in dispatch, while unrelated launch keys survive. + let cases = [ + ("empty", vec![], "baml", 0, false), + ("help", vec!["--help"], "baml", 0, false), + ( + "help-stop", + vec!["--help", "toolchain", "update"], + "baml", + 0, + false, + ), + ("version", vec!["--version"], "baml", 0, false), + ("check", vec!["check"], "baml check", 0, false), + ( + "delegated-error", + vec!["check", "--private-marker-secret"], + "baml check", + 2, + false, + ), + ( + "global-project", + vec!["--project", project_path, "check"], + "baml check", + 0, + false, + ), + ( + "option-value", + vec!["--color", "never", "check"], + "baml check", + 0, + false, + ), + ( + "query", + vec!["query", "--project", "private-marker-secret", "--help"], + "baml query", + 0, + false, + ), + ("unknown", vec!["private-marker-secret"], "baml", 2, false), + ( + "terminator", + vec!["--", "toolchain", "update"], + "baml", + 2, + false, + ), + ( + "toolchain-help", + vec!["toolchain", "--help"], + "baml toolchain", + 0, + false, + ), + ( + "nested-help", + vec!["help", "toolchain", "update"], + "baml toolchain update", + 0, + false, + ), + ( + "use-help", + vec!["toolchain", "use", "--help"], + "baml toolchain use", + 0, + false, + ), + ( + "status", + vec!["toolchain", "status", "--json"], + "baml toolchain status", + 0, + false, + ), + ( + "install-help", + vec!["toolchain", "install", "--help"], + "baml toolchain install", + 0, + false, + ), + ( + "update-help", + vec!["toolchain", "update", "--help"], + "baml toolchain update", + 0, + false, + ), + ( + "list", + vec!["toolchain", "list"], + "baml toolchain list", + 0, + false, + ), + ( + "uninstall-help", + vec!["toolchain", "uninstall", "--help"], + "baml toolchain uninstall", + 0, + false, + ), + ("parse-error", vec!["--project"], "baml", 1, true), + ("selection-error", vec!["check"], "baml check", 1, true), + ( + "project-selection-error", + vec!["toolchain", "use", "./private-marker-secret"], + "baml toolchain use", + 1, + true, + ), + ]; + for (scenario, args, _, code, _) in &cases { + let output = local_command(&binary, &home) + .args(args) + .current_dir(&project) + .env("BAML_TELEMETRY", "medium") + .env( + "BAML_CONTEXT", + if *scenario == "check" { + json!({"metadata": {"scenario": scenario, "command": "caller-command"}}) + } else { + json!({"metadata": {"scenario": scenario}}) + } + .to_string(), + ) + .env( + "BAML_TOOLCHAIN", + if *scenario == "selection-error" { + Path::new("invalid-version") + } else { + &built.baml_cli + }, + ) + .output() + .unwrap(); + assert_eq!(output.status.code(), Some(*code), "{scenario}: {output:?}"); + } + // These helper invocations must clear an inherited foreground command. + for (scenario, args, code) in [ + ("flush-helper", vec!["__flush-wrapper-telemetry"], 0), + ("replace-helper", vec!["--replace"], 1), + ("recording-off", vec!["--help"], 0), + ] { + let output = local_command(&binary, &home) + .args(args) + .current_dir(&project) + .env( + "BAML_TELEMETRY", + if scenario == "recording-off" { + "off" + } else { + "medium" + }, + ) + .env( + "BAML_CONTEXT", + json!({"metadata": {"scenario": scenario, "command": "caller-command"}}) + .to_string(), + ) + .output() + .unwrap(); + assert_eq!(output.status.code(), Some(code), "{scenario}: {output:?}"); + } + let query = |sql: &str| { + let output = local_command(&built.baml_cli, &home) + .args(["query", "--local", "--format", "json", "--from"]) + .arg(temp.path().join("home")) + .arg(sql) + .output() + .unwrap(); + succeeded(&output); + serde_json::from_slice::(&output.stdout).unwrap()["rows"].take() + }; + // Exactly one dispatch per foreground process: no multiply-counting the + // inherited parser/management spans. sys.exit cancels delegated dispatch, + // so process status, not dispatch status, supplies the invocation outcome. + let rows = query( + "SELECT s.context_metadata['scenario'], s.context_metadata['command'], + p.status, s.status, COUNT(*) + FROM spans s JOIN processes p ON s.process_id = p.process_id + WHERE s.span_name = 'user.dispatch' + GROUP BY s.context_metadata['scenario'], s.context_metadata['command'], p.status, s.status + ORDER BY s.context_metadata['scenario']", + ); + let mut expected = cases + .iter() + .map(|(scenario, _, command, code, wrapper_error)| { + json!([ + scenario, + command, + if *code == 0 { "success" } else { "error" }, + if *wrapper_error { + "user_error" + } else if [ + "empty", + "help", + "toolchain-help", + "nested-help", + "use-help", + "status", + "install-help", + "update-help", + "list", + "uninstall-help" + ] + .contains(scenario) + { + "return" + } else { + "cancel_error" + }, + 1 + ]) + }) + .collect::>(); + expected.push(json!(["flush-helper", null, "success", "return", 1])); + expected.push(json!(["replace-helper", null, "error", "user_error", 1])); + expected.sort_by(|a, b| a[0].as_str().cmp(&b[0].as_str())); + assert_eq!(rows, json!(expected)); + + for relation in ["spans", "span_announcements"] { + assert_eq!( + query(&format!( + "SELECT COUNT(*) FROM {relation} WHERE span_name = 'user.WrapperInvocation.parse'" + )), + json!([[cases.len()]]), + "{relation}: each foreground invocation reaches parsing, including parse failures" + ); + assert_eq!( + query(&format!( + "SELECT COUNT(*) FROM {relation} WHERE span_name = 'user.dispatch'" + )), + json!([[cases.len() + 2]]), + "{relation}: exactly one dispatch per recorded wrapper invocation" + ); + assert_eq!( + query(&format!( + "SELECT COUNT(*) FROM {relation} s JOIN spans d ON s.process_id = d.process_id + WHERE d.span_name = 'user.dispatch' AND s.span_name = 'user.WrapperInvocation.parse' + AND (s.context_metadata['command'] != d.context_metadata['command'] + OR baml_value_state(s.context_metadata['command']) != 'present')" + )), + json!([[0]]), + "{relation}: parser inherits dispatch context" + ); + assert_eq!( + query(&format!( + "SELECT COUNT(*) FROM {relation} WHERE span_name = 'user.dispatch' + AND context_metadata['command'] LIKE '%private-marker-secret%'" + )), + json!([[0]]) + ); + assert_eq!( + query(&format!( + "SELECT COUNT(*) FROM {relation} WHERE span_name = 'user.dispatch' + AND context_metadata['command'] LIKE '%--%'" + )), + json!([[0]]) + ); + } + assert_eq!( + query( + "SELECT COUNT(*) FROM processes WHERE context_metadata['scenario'] = 'recording-off'" + ), + json!([[0]]) + ); + assert_eq!( + query( + "SELECT COUNT(*) FROM processes WHERE context_metadata['command'] = 'caller-command'" + ), + json!([[3]]) + ); + assert_eq!( + query( + "SELECT COUNT(*) FROM processes WHERE baml_value_state(context_metadata['command']) = 'present'" + ), + json!([[3]]), + "dispatch context never replaces or adds process launch context" + ); + assert_eq!(query("SELECT COUNT(*) FROM spans WHERE span_type = 'function' + AND span_name IN ('user.dispatch', 'user.WrapperInvocation.parse', 'user.management_command', + 'user.WrapperInvocation.selection_impl', 'user.WrapperUpdater.update', 'user.run_toolchain') + AND baml_value_state(input_args) != 'no_value'"), json!([[0]])); + assert_eq!(query("SELECT COUNT(*) FROM spans WHERE span_type = 'function' + AND span_name IN ('user.dispatch', 'user.WrapperInvocation.parse', 'user.management_command', + 'user.WrapperInvocation.selection_impl', 'user.WrapperUpdater.update') + AND baml_value_state(output_value) != 'no_value'"), json!([[0]])); + assert_eq!( + query( + "SELECT COUNT(*) FROM spans WHERE span_name = 'user.WrapperInvocation.parse' + AND context_metadata['scenario'] = 'parse-error' AND status = 'user_error' + AND baml_value_state(error_value) = 'present'" + ), + json!([[1]]) + ); + assert!( + !home.join("wrapper-telemetry.toml").exists(), + "PostHog collection stayed disabled" + ); + assert!( + !home.join("wrapper-telemetry").exists(), + "PostHog delivery stayed disabled" + ); +} diff --git a/baml_language/wrapper/README.md b/baml_language/wrapper/README.md index 9219ec767c6..2af526744ef 100644 --- a/baml_language/wrapper/README.md +++ b/baml_language/wrapper/README.md @@ -2,7 +2,32 @@ This is the standalone BAML implementation of the `baml` toolchain wrapper. It r The wrapper uses the process APIs adopted in #5456 and the filesystem, path, archive, atomic-write, locking, and typed TOML implementations adopted in #5457. Host telemetry metadata and strict archive preflight are included in #5457. Subprocess execution retains normal platform signal delivery without automatic parent-to-child signal forwarding. -BCS retains explicit spans for dispatch, argument parsing, management command preflight/execution, and self-update. These spans capture no arguments, outputs, or error payloads. Counting argument-parser spans excludes detached telemetry/replacement helpers; update handler spans exclude command help. An error escaping dispatch is a wrapper failure, while recovered internal errors and delegated CLI exit codes are excluded. These boundaries remain queryable independently of completed aggregate profiles. +BCS retains explicit operational spans with canonical command context and error payloads. Full inputs and argument-bearing returned objects are excluded: + +| Boundary | Captured values | +| --- | --- | +| `WrapperInvocation.parse`, `management_command`, `WrapperInvocation.selection_impl` | Thrown errors; no argv, parsed command, project/working-directory or selector result snapshots. | +| Toolchain acquisition, channel freshness/refresh implementations, management `preflight` / `execute`, `WrapperUpdater.update`, `dispatch` | Error payloads; no input or returned-value snapshots, including null or updater-copy paths. | +| `run_toolchain` | Only returned `baml.sys.ProcessExit { exit_code, signal }` and launch errors. The selected binary and forwarded arguments are not captured. Version reporting uses this boundary too. | + +Child stdout/stderr remain inherited and uncaptured. A delegated CLI/project keeps its own recording policy. Functions unwound by `sys.exit` have no returned output; the child-status boundary preserves exit code/signal before that unwind. Full errors are intentionally recorded and may quote offending arguments, selectors or paths, including secrets. Omitting input/result snapshots does not redact error payloads. PostHog sender objects and internal sender payloads are excluded from inputs/results; no identities or destinations are added. + +Internal telemetry/replacement helpers bypass parsing. Management and self-update help bypass their handlers. Acquisition and passive freshness/refresh implementation spans retain errors before their existing recovery handlers. Errors escaping a traced boundary remain visible even if its caller recovers; errors recovered entirely within an untraced utility retain its automatic policy. Panic errors and cancellation keep their runtime classifications. Explicit capture applies only to the requested invocation, not descendants; `high` promotes automatic function spans without enabling additional value capture. Packed `BAML_TELEMETRY=on` retains the baked level (normally medium), while `off` disables recording, including explicit spans. HTTP bodies remain excluded by `http_bodies = false`. This BCS policy is separate from bounded PostHog usage events and the automatic cloud OS metadata collector, which omits argv. + +The foreground `user.dispatch` span carries `context_metadata['command']`, computed by the same `WrapperTelemetry.command(args)` normalization used for PostHog. It is attached before parsing and inherited by parser, management, and other descendant spans, independently of argument capture and PostHog collection/delivery. Values remain bounded canonical labels such as `baml check`, `baml query`, and `baml toolchain use`; no flags or argument values enter this metadata. The label `baml` is ambiguous: empty argv, root help, version, unknown commands, and `--` termination can all normalize to it. Internal telemetry and replacement dispatches remove the foreground `command` key, including a caller-supplied launch value. + +This is invocation-span context, not process launch context: it does not populate `processes.context_metadata['command']` or `processes.command`. Join the one `user.dispatch` row per invocation to `processes` using `process_id` to group commands with process outcomes, without raw argv or PostHog joins. Delegated exits can cancel dispatch even when the process succeeds, so use the process status for invocation outcomes; dispatch `user_error` identifies errors escaping the wrapper handler. Recording must be enabled; `BAML_TELEMETRY=off` disables local sinks too. For an isolated local run, use `DO_NOT_TRACK=1 BOUNDARY_API_KEY=local BAML_TELEMETRY=medium BAML_HOME=/tmp/wrapper-command-home/.baml /path/to/baml toolchain list`, then query with: + +```sh +DO_NOT_TRACK=1 BOUNDARY_API_KEY=local BAML_TELEMETRY=off /path/to/baml-cli query --local --from /tmp/wrapper-command-home --format json \ + "SELECT s.context_metadata['command'] AS command, p.status, COUNT(*) AS invocations + FROM spans s JOIN processes p ON s.process_id = p.process_id + WHERE s.span_name = 'user.dispatch' + AND baml_value_state(s.context_metadata['command']) = 'present' + GROUP BY s.context_metadata['command'], p.status" +``` + +For a cloud query, use the same SQL against the wrapper recording project. Select only dispatch rows when counting invocations: descendant spans inherit the command, and `span_announcements` exposes it for announced spans before completion, but counting both relations or every descendant would duplicate invocations. A recording lost before dispatch is announced cannot supply the label. No shared end-user identity is added. The code is organized around the data each operation owns: `WrapperInvocation` parses arguments and coordinates selection/management; version, channel, and path selectors implement their own behavior through `ToolchainSelector`; command classes carry only their applicable options. Configuration constructors validate documents before writes, and `ReleaseStore` keeps a validated release source with its cache. `WrapperTelemetry` owns a session and `TelemetryQueue` distinguishes delivered, rejected, and unavailable outcomes. Recoverable I/O/configuration failures are handled at the operation that can recover; the entry point renders remaining errors. Shared wrapper tests use descriptive `test` blocks with their setup and assertions in one place. diff --git a/baml_language/wrapper/baml_src/entry.baml b/baml_language/wrapper/baml_src/entry.baml index 9d2bc937585..e9d43255ca8 100644 --- a/baml_language/wrapper/baml_src/entry.baml +++ b/baml_language/wrapper/baml_src/entry.baml @@ -103,9 +103,21 @@ function main() -> void throws never { // scheduling another helper that immediately retries the same rejection. if (!internal_helper) { WrapperTelemetry.finish(telemetry, executable = flush_executable); } } - // Retain command outcomes without recording arguments or error payloads. + // Capture canonical command context and errors without argument snapshots. + // Helpers must not inherit a caller-supplied foreground command label. flush_executable - = dispatch(telemetry, $trace = trace.span(inputs = false, output = false, error = false)) catch (error) { + = dispatch( + telemetry, + $trace = trace.span(inputs = false, output = false, error = true).context( + metadata = { + "command": if (internal_helper) { + null + } else { + WrapperTelemetry.command(args) + }, + }, + ), + ) catch (error) { _ => { telemetry_event(telemetry, "wrapper_unhandled_error", fields = { "event_kind": "unhandled_error" }); baml.io.eprintln(`error: ${error_message(error)}`); @@ -129,7 +141,7 @@ function dispatch(telemetry: WrapperTelemetry?) -> string? { } let paths = WrapperInvocation.parse( forwarded, - $trace = trace.span(inputs = false, output = false, error = false), + $trace = trace.span(inputs = false, output = false, error = true), ); let args = paths.args; paths.telemetry = telemetry; @@ -163,7 +175,7 @@ function dispatch(telemetry: WrapperTelemetry?) -> string? { ensure(args.length() == 1, "unexpected self-update arguments"); return WrapperUpdater.update( telemetry = paths.telemetry, - $trace = trace.span(inputs = false, output = false, error = false), + $trace = trace.span(inputs = false, output = false, error = true), ); }, _ => {}, @@ -173,7 +185,7 @@ function dispatch(telemetry: WrapperTelemetry?) -> string? { let warned = active.warn_if_outdated(); let refresh = spawn { active.refresh_channel() }; WrapperTelemetry.finish(paths.telemetry); - let result = baml.sys.run(binary, args = forwarded, env = { "BAML_CLI_ALLOW_DIRECT": "1" }); + let result = run_toolchain(binary, forwarded, $trace = trace.span(inputs = false, output = true, error = true)); await refresh; if (!warned) { active.warn_if_outdated(); @@ -181,6 +193,13 @@ function dispatch(telemetry: WrapperTelemetry?) -> string? { baml.sys.exit(result.exit_code) } +// Keep the child result before sys.exit unwinds dispatch as cancellation. Standard +// streams still inherit; expose only status to the recording boundary. +function run_toolchain(binary: string, args: string[]) -> baml.sys.ProcessExit { + let result = baml.sys.run(binary, args = args, env = { "BAML_CLI_ALLOW_DIRECT": "1" }); + baml.sys.ProcessExit { exit_code: result.exit_code, signal: result.signal } +} + function describe_selector(value: string) -> string { if (channel(value)) { let active = ChannelState.new().recorded(value) catch (error) { diff --git a/baml_language/wrapper/baml_src/invocation.baml b/baml_language/wrapper/baml_src/invocation.baml index 3a5fda58f34..33dcb2d0923 100644 --- a/baml_language/wrapper/baml_src/invocation.baml +++ b/baml_language/wrapper/baml_src/invocation.baml @@ -5,7 +5,7 @@ class WrapperInvocation { project: string, function selection(self) -> Selection { - let active = self.selection_impl() catch (error) { + let active = self.selection_impl($trace = trace.span(inputs = false, output = false, error = true)) catch (error) { _ => { telemetry_event(self.telemetry, "wrapper_resolution", fields = { "outcome": "failure" }); throw error; @@ -326,7 +326,7 @@ class WrapperInvocation { function run_cli_version(self, flag: string) -> int { let binary = self.selection().ensure_installed(telemetry = self.telemetry); WrapperTelemetry.finish(self.telemetry); - let result = baml.sys.run(binary, args = [flag], env = { "BAML_CLI_ALLOW_DIRECT": "1" }); + let result = run_toolchain(binary, [flag], $trace = trace.span(inputs = false, output = true, error = true)); result.exit_code } @@ -353,9 +353,11 @@ class WrapperInvocation { help(command = args.at(1) ?? ""); return null; } - let command = management_command(args); - command.preflight(self, $trace = trace.span(inputs = false, output = false, error = false)); - command.execute(self, $trace = trace.span(inputs = false, output = false, error = false)) + // Keep errors queryable without recording argv, parsed options, or the + // invocation's telemetry sender through inputs or returned objects. + let command = management_command(args, $trace = trace.span(inputs = false, output = false, error = true)); + command.preflight(self, $trace = trace.span(inputs = false, output = false, error = true)); + command.execute(self, $trace = trace.span(inputs = false, output = false, error = true)) } function parse(raw: string[]) -> WrapperInvocation { diff --git a/baml_language/wrapper/baml_src/selection.baml b/baml_language/wrapper/baml_src/selection.baml index 507c5dfbbb6..53257d15b18 100644 --- a/baml_language/wrapper/baml_src/selection.baml +++ b/baml_language/wrapper/baml_src/selection.baml @@ -17,7 +17,11 @@ class Selection { } function ensure_installed(self, telemetry: WrapperTelemetry? = null) -> string { - self.selector.ensure_installed(self.source, telemetry) + self.selector.ensure_installed( + self.source, + telemetry, + $trace = trace.span(inputs = false, output = false, error = true), + ) } function warn_if_outdated(self) -> bool { diff --git a/baml_language/wrapper/baml_src/selectors.baml b/baml_language/wrapper/baml_src/selectors.baml index 3b0f1e20960..ab370a81e04 100644 --- a/baml_language/wrapper/baml_src/selectors.baml +++ b/baml_language/wrapper/baml_src/selectors.baml @@ -143,13 +143,17 @@ class ChannelSelector { } function warn_if_outdated(self) -> bool { - self.warn_if_outdated_impl() catch (error) { + self.warn_if_outdated_impl( + $trace = trace.span(inputs = false, output = false, error = true), + ) catch (error) { WrapperFailure => false } } function refresh_channel(self) -> null { - self.refresh_channel_impl() catch (error) { + self.refresh_channel_impl( + $trace = trace.span(inputs = false, output = false, error = true), + ) catch (error) { WrapperFailure => null } } From 3289a1ab8e4f39368a9eac6e0c7a09ad08ef1741 Mon Sep 17 00:00:00 2001 From: Sam Lijin Date: Thu, 12 Nov 2026 15:41:17 -0800 Subject: [PATCH 2/3] Test retained child termination signal in wrapper recordings --- .../baml_cli/tests/embedded_telemetry_e2e.rs | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/baml_language/crates/baml_cli/tests/embedded_telemetry_e2e.rs b/baml_language/crates/baml_cli/tests/embedded_telemetry_e2e.rs index 0e796a76f85..e761cbaf5c7 100644 --- a/baml_language/crates/baml_cli/tests/embedded_telemetry_e2e.rs +++ b/baml_language/crates/baml_cli/tests/embedded_telemetry_e2e.rs @@ -951,6 +951,20 @@ function main() -> void { #[cfg(unix)] { use std::os::unix::fs::PermissionsExt as _; + let signaled_child = temp.path().join("signaled-child"); + std::fs::write(&signaled_child, "#!/bin/sh\nkill -TERM $$\n").unwrap(); + std::fs::set_permissions(&signaled_child, std::fs::Permissions::from_mode(0o755)).unwrap(); + let output = local(&binary) + .arg("check") + .env("BAML_TOOLCHAIN", &signaled_child) + .env("BAML_TELEMETRY", "medium") + .env( + "BAML_CONTEXT", + json!({"metadata": {"scenario": "signal"}}).to_string(), + ) + .output() + .unwrap(); + assert_eq!(output.status.code(), Some(143), "{output:?}"); let invalid_binary = temp.path().join("launch-failure-sentinel"); // A missing shebang interpreter guarantees a spawn error; a plain // invalid executable may be launched through the platform shell. @@ -1240,6 +1254,20 @@ function main() -> void { ); #[cfg(unix)] { + let signaled = span("signal", "user.run_toolchain"); + assert_eq!(signaled[0], "return"); + assert_eq!(signaled[2]["$class"], "baml.sys.ProcessExit"); + assert_eq!(signaled[2]["exit_code"], 143); + assert_eq!(signaled[2]["signal"], 15); + assert_eq!(signaled[4], "no_value"); + assert_eq!(signaled[7], "baml check"); + assert_eq!(span("signal", "user.dispatch")[0], "cancel_error"); + assert_eq!( + query( + "SELECT DISTINCT status FROM processes WHERE context_metadata['scenario'] = 'signal'" + ), + json!([["error"]]) + ); let panicked = span("panic", "user.dispatch"); assert_eq!(panicked[0], "panic_error"); assert_eq!( From 049b4f221548a66b1d09557a993051c5edcaa757 Mon Sep 17 00:00:00 2001 From: Sam Lijin Date: Thu, 12 Nov 2026 16:15:20 -0800 Subject: [PATCH 3/3] Preserve capture across requested-target acquisition changes --- TELEMETRY.md | 2 +- .../baml_cli/tests/embedded_telemetry_e2e.rs | 75 +++++++++++++++---- baml_language/wrapper/README.md | 4 +- .../wrapper/baml_src/invocation.baml | 5 +- 4 files changed, 67 insertions(+), 19 deletions(-) diff --git a/TELEMETRY.md b/TELEMETRY.md index f2961326f4d..052836f531d 100644 --- a/TELEMETRY.md +++ b/TELEMETRY.md @@ -94,7 +94,7 @@ Wrapper events are written locally before a toolchain handoff. If a handoff fail The Rust wrapper sends only PostHog usage events. Release builds of the BAML-native wrapper additionally enable BCS runtime recording through an embedded public ingestion credential. BCS uses the existing packed-runtime policy: delivery runs on a background worker and shutdown waits for its normal bounded drain. The wrapper selects the existing `on_initial_telemetry_failure = "ignore"` policy. The PostHog queue retains its separate retry behavior. Set `BAML_TELEMETRY=off` to disable the BAML wrapper’s BCS recording. -The BAML-native wrapper records error payloads at its explicit dispatch, argument-parser, management-parser/preflight/execution, toolchain-selection/acquisition, passive freshness/refresh and self-update boundaries. It does not request full input snapshots or argument-bearing returned objects. Delegated and version calls retain only child exit code and signal in a `baml.sys.ProcessExit`; child stdout/stderr remain inherited and uncaptured. A delegated CLI/project keeps its own recording policy. Functions unwound by `sys.exit` have no returned output. Error payloads are retained as supplied and can quote offending arguments, selectors or paths, including secrets; suppressing input snapshots does not redact errors. Internal telemetry/replacement helpers bypass argument parsing, and command help bypasses management parsing and self-update. Capture applies to the requested invocation, not all descendants; errors recovered entirely inside an untraced utility retain its automatic policy. Other automatic functions do not acquire value capture from these requests or from `high`. For the packed wrapper, `BAML_TELEMETRY=on` keeps the baked default level (normally medium), while `off` disables BCS recording. HTTP network spans still exclude bodies. See the [wrapper capture boundary table](baml_language/wrapper/README.md) for exact scopes. This BCS policy does not change PostHog's metadata-only contract, identities, destinations or existing launch metadata. +The BAML-native wrapper records error payloads at its explicit dispatch, argument-parser, management-parser/preflight/execution, toolchain-selection/acquisition, selected-CLI version reporting, passive freshness/refresh and self-update boundaries. It does not request full input snapshots or argument-bearing returned objects. Delegated and version calls retain only child exit code and signal in a `baml.sys.ProcessExit`; child stdout/stderr remain inherited and uncaptured. A delegated CLI/project keeps its own recording policy. Functions unwound by `sys.exit` have no returned output. Error payloads are retained as supplied and can quote offending arguments, selectors or paths, including secrets; suppressing input snapshots does not redact errors. Internal telemetry/replacement helpers bypass argument parsing, and command help bypasses management parsing and self-update. Capture applies to the requested invocation, not all descendants; errors recovered entirely inside an untraced utility retain its automatic policy. Other automatic functions do not acquire value capture from these requests or from `high`. For the packed wrapper, `BAML_TELEMETRY=on` keeps the baked default level (normally medium), while `off` disables BCS recording. HTTP network spans still exclude bodies. See the [wrapper capture boundary table](baml_language/wrapper/README.md) for exact scopes. This BCS policy does not change PostHog's metadata-only contract, identities, destinations or existing launch metadata. The BAML-native wrapper also attaches the same bounded canonical `command` to the foreground `user.dispatch` span and descendants, independently of PostHog and function value capture. Helpers remove this key in their dispatch scope. This does not update process launch metadata or add an identity; join dispatch to `processes` by `process_id` for process outcomes. See the [wrapper query example](baml_language/wrapper/README.md) for counting invocations without duplicating inherited spans. diff --git a/baml_language/crates/baml_cli/tests/embedded_telemetry_e2e.rs b/baml_language/crates/baml_cli/tests/embedded_telemetry_e2e.rs index e761cbaf5c7..cdd6a5923bd 100644 --- a/baml_language/crates/baml_cli/tests/embedded_telemetry_e2e.rs +++ b/baml_language/crates/baml_cli/tests/embedded_telemetry_e2e.rs @@ -911,6 +911,8 @@ function main() -> void { ("helper", vec!["__flush-wrapper-telemetry"], 0), ("delegated", vec!["check", "delegated-input-sentinel"], 7), ("version", vec!["--version"], 7), + ("version-acquisition-error", vec!["--version"], 0), + ("version-selection-error", vec!["--version"], 0), ("passive-errors", vec!["check", "exit-zero"], 0), ("high", vec!["--help"], 0), ("on", vec!["--help"], 0), @@ -918,7 +920,7 @@ function main() -> void { ] { let toolchain = match scenario { "delegated" | "version" => child.to_str().unwrap(), - "list" => "0.0.78", + "list" | "version-acquisition-error" => "0.0.78", "passive-errors" => "canary", _ => "invalid-version-sentinel", }; @@ -974,17 +976,22 @@ function main() -> void { ) .unwrap(); std::fs::set_permissions(&invalid_binary, std::fs::Permissions::from_mode(0o755)).unwrap(); - let output = local(&binary) - .args(["check", "launch-input-sentinel"]) - .env("BAML_TOOLCHAIN", &invalid_binary) - .env("BAML_TELEMETRY", "medium") - .env( - "BAML_CONTEXT", - json!({"metadata": {"scenario": "launch-error"}}).to_string(), - ) - .output() - .unwrap(); - assert_eq!(output.status.code(), Some(1), "{output:?}"); + for (scenario, args, code) in [ + ("launch-error", vec!["check", "launch-input-sentinel"], 1), + ("version-launch-error", vec!["--version"], 0), + ] { + let output = local(&binary) + .args(args) + .env("BAML_TOOLCHAIN", &invalid_binary) + .env("BAML_TELEMETRY", "medium") + .env( + "BAML_CONTEXT", + json!({"metadata": {"scenario": scenario}}).to_string(), + ) + .output() + .unwrap(); + assert_eq!(output.status.code(), Some(code), "{scenario}: {output:?}"); + } } #[cfg(unix)] { @@ -1122,10 +1129,17 @@ function main() -> void { assert_eq!( span("list", "user.dispatch")[0], "return", - "recovered preflight failures stay recovered" + "local inventory needs no acquisition" + ); + assert_eq!( + query( + "SELECT COUNT(*) FROM spans WHERE context_metadata['scenario'] = 'list' + AND span_name LIKE '%ensure_installed%'" + ), + json!([[0]]) ); let acquired = span( - "list", + "version-acquisition-error", "user.<(user.VersionSelector as user.ToolchainSelector)>.ensure_installed", ); assert_eq!( @@ -1144,7 +1158,21 @@ function main() -> void { ); assert_eq!(acquired[4], "no_value"); assert_eq!(acquired[5], "no_value"); - assert_eq!(acquired[7], "baml toolchain list"); + assert_eq!(acquired[7], "baml"); + for scenario in ["version-acquisition-error", "version-selection-error"] { + let reported = span(scenario, "user.WrapperInvocation.run_cli_version"); + assert_eq!(reported[0], "user_error"); + assert_eq!(reported[4], "no_value"); + assert_eq!(reported[5], "no_value"); + assert_eq!(reported[6], "present"); + assert_eq!(span(scenario, "user.dispatch")[0], "cancel_error"); + assert_eq!( + query(&format!( + "SELECT DISTINCT status FROM processes WHERE context_metadata['scenario'] = '{scenario}'" + )), + json!([["success"]]) + ); + } assert_eq!( span( "list", @@ -1288,6 +1316,23 @@ function main() -> void { assert_eq!(launch[4], "no_value"); assert_eq!(launch[5], "no_value"); assert_eq!(launch[7], "baml check"); + let recovered_launch = span("version-launch-error", "user.run_toolchain"); + assert_eq!(recovered_launch[0], "user_error"); + assert_eq!(recovered_launch[3]["error"]["$class"], "baml.errors.Io"); + assert_eq!(recovered_launch[5], "no_value"); + assert_eq!( + span( + "version-launch-error", + "user.WrapperInvocation.run_cli_version" + )[0], + "user_error" + ); + assert_eq!( + query( + "SELECT DISTINCT status FROM processes WHERE context_metadata['scenario'] = 'version-launch-error'" + ), + json!([["success"]]) + ); } let automatic = span("high", "user.root_help"); assert_eq!(automatic[0], "return"); diff --git a/baml_language/wrapper/README.md b/baml_language/wrapper/README.md index 2af526744ef..46261fa093d 100644 --- a/baml_language/wrapper/README.md +++ b/baml_language/wrapper/README.md @@ -7,12 +7,12 @@ BCS retains explicit operational spans with canonical command context and error | Boundary | Captured values | | --- | --- | | `WrapperInvocation.parse`, `management_command`, `WrapperInvocation.selection_impl` | Thrown errors; no argv, parsed command, project/working-directory or selector result snapshots. | -| Toolchain acquisition, channel freshness/refresh implementations, management `preflight` / `execute`, `WrapperUpdater.update`, `dispatch` | Error payloads; no input or returned-value snapshots, including null or updater-copy paths. | +| Toolchain acquisition, channel freshness/refresh implementations, selected-CLI version reporting, management `preflight` / `execute`, `WrapperUpdater.update`, `dispatch` | Error payloads; no input or returned-value snapshots, including null or updater-copy paths. | | `run_toolchain` | Only returned `baml.sys.ProcessExit { exit_code, signal }` and launch errors. The selected binary and forwarded arguments are not captured. Version reporting uses this boundary too. | Child stdout/stderr remain inherited and uncaptured. A delegated CLI/project keeps its own recording policy. Functions unwound by `sys.exit` have no returned output; the child-status boundary preserves exit code/signal before that unwind. Full errors are intentionally recorded and may quote offending arguments, selectors or paths, including secrets. Omitting input/result snapshots does not redact error payloads. PostHog sender objects and internal sender payloads are excluded from inputs/results; no identities or destinations are added. -Internal telemetry/replacement helpers bypass parsing. Management and self-update help bypass their handlers. Acquisition and passive freshness/refresh implementation spans retain errors before their existing recovery handlers. Errors escaping a traced boundary remain visible even if its caller recovers; errors recovered entirely within an untraced utility retain its automatic policy. Panic errors and cancellation keep their runtime classifications. Explicit capture applies only to the requested invocation, not descendants; `high` promotes automatic function spans without enabling additional value capture. Packed `BAML_TELEMETRY=on` retains the baked level (normally medium), while `off` disables recording, including explicit spans. HTTP bodies remain excluded by `http_bodies = false`. This BCS policy is separate from bounded PostHog usage events and the automatic cloud OS metadata collector, which omits argv. +Internal telemetry/replacement helpers bypass parsing. Management and self-update help bypass their handlers. Acquisition, selected-CLI version reporting, and passive freshness/refresh implementation spans retain errors before their existing recovery handlers. Version preparation or launch failures remain advisory and leave wrapper version reporting successful; a child that runs still propagates its exit code. Errors escaping a traced boundary remain visible even if its caller recovers; errors recovered entirely within an untraced utility retain its automatic policy. Panic errors and cancellation keep their runtime classifications. Explicit capture applies only to the requested invocation, not descendants; `high` promotes automatic function spans without enabling additional value capture. Packed `BAML_TELEMETRY=on` retains the baked level (normally medium), while `off` disables recording, including explicit spans. HTTP bodies remain excluded by `http_bodies = false`. This BCS policy is separate from bounded PostHog usage events and the automatic cloud OS metadata collector, which omits argv. The foreground `user.dispatch` span carries `context_metadata['command']`, computed by the same `WrapperTelemetry.command(args)` normalization used for PostHog. It is attached before parsing and inherited by parser, management, and other descendant spans, independently of argument capture and PostHog collection/delivery. Values remain bounded canonical labels such as `baml check`, `baml query`, and `baml toolchain use`; no flags or argument values enter this metadata. The label `baml` is ambiguous: empty argv, root help, version, unknown commands, and `--` termination can all normalize to it. Internal telemetry and replacement dispatches remove the foreground `command` key, including a caller-supplied launch value. diff --git a/baml_language/wrapper/baml_src/invocation.baml b/baml_language/wrapper/baml_src/invocation.baml index 33dcb2d0923..b956c21a4d8 100644 --- a/baml_language/wrapper/baml_src/invocation.baml +++ b/baml_language/wrapper/baml_src/invocation.baml @@ -312,7 +312,10 @@ class WrapperInvocation { function print_version(self, flag: string) -> null { baml.io.println(`baml wrapper ${wrapper_version()}`); - let exit_code = self.run_cli_version(flag) catch (error) { + let exit_code = self.run_cli_version( + flag, + $trace = trace.span(inputs = false, output = false, error = true), + ) catch (error) { WrapperFailure => { baml.io.eprintln( `warning: could not report selected baml-cli version: ${error_message(error)}`,