fix: share fixed per-project trace stores under BAML_HOME

baml_language/ENV_VARS.md

@@ -34,15 +34,15 @@ Windows keeps an empty `login/cache/<endpoint-hash>.lock` beside each credential
 
 | Variable | What it does |
 |---|---|
-| `BAML_HOME` | Move all BAML state (CI workspace, container, read-only home). |
+| `BAML_HOME` | Move user state (CI workspace, container, read-only home), including per-project local trace stores; see below. Empty means unset. |
 | `BAML_TOOLCHAIN` | Which toolchain `baml` runs: a channel, a version, or a path to a local `baml-cli`. Default `canary`. |
 | `BAML_MANIFEST_BASE_URL` | Release manifest mirror, used by the wrapper and the bridges. |
 | `BAML_BUILD_CACHE` | `false` turns off the build cache. |
 | `BAML_BUILD_CACHE_REMOTE`, `BAML_BUILD_CACHE_REMOTE_TOKEN` | Shared build cache for CI. URL must be https (or localhost). |
 | `BAML_LOG` | Level for `log.*` output from `baml run` and `baml test` (same as `--log`): `off`, `error`, `warn`, `info` (default), `debug`, `trace`. |
 | `BAML_TELEMETRY` | Runtime tracing level: `off`, `low`, `medium` (default), `high`. |
 | `BOUNDARY_API_URL` | Boundary API gateway for login, cloud queries and telemetry. Overrides `[boundary].api_url`; default `https://api.prod.bcs.boundaryml.com`. |
-| `BOUNDARY_API_KEY` | Non-interactive Boundary credential for cloud queries and telemetry. Takes precedence over saved user login. |
+| `BOUNDARY_API_KEY` | Non-interactive Boundary credential for cloud queries and telemetry. Takes precedence over saved user login. Set to `local` to record and query locally even when a saved cloud login exists. |
 | `BOUNDARY_PROJECT` | Cloud target as `org_handle/project_name`. Overrides `[boundary].project`; query's `--project` flag takes precedence. |
 | `BAML_CLI_ALLOW_DIRECT` | Hides the "don't run baml-cli directly" warning. The wrapper sets it. |
 | `BAML_FEEDBACK_HOST` | Where `baml feedback` sends reports. Staging and tests. |
@@ -80,3 +80,44 @@ Go, Rust, C++, Java, C# and TypeScript load a native library. Python and Swift l
 `NAPI_RS_NATIVE_LIBRARY_PATH`, `NAPI_RS_FORCE_WASI` and `NAPI_RS_ENFORCE_VERSION_CHECK` belong to the generated napi-rs loader, not to us.
 
 Not listed: test, benchmark and CI-only variables, Ruby, and the old `engine/`.
+
+## Shared local trace store
+
+SDK bridge recordings, playground runs, `baml run` / `baml test`, and `baml query --local` share a fixed per-project store at `<BAML_HOME>/<project>`. `BAML_HOME` is the BAML state directory itself, defaulting to `~/.baml`; no extra `.baml` is appended. No trace-store setting is needed or supported in `baml.toml`, and recording does not create `<project-root>/.baml`.
+
+`<project>` is the project directory's name (sanitized and limited to 64 characters), followed by a hyphen and the first 16 hexadecimal digits of the SHA-256 of its canonical filesystem path. Different checkouts with the same name have separate stores; symlinked paths to the same checkout share a store. Moving a project changes its store identity. Each store contains `btel/recordings/`, shared `btel/cas/` blobs, and the disposable `btel/query.sqlite` index. Store resolution reads no manifest settings and creates no directories until recording or indexing needs them. If neither a nonempty `BAML_HOME` nor an OS home directory is available, the base falls back to `.baml` relative to the working directory.
+
+The CLI uses its resolved project root, including an explicit project passed to `--from`. The SDK bridge discovers the project enclosing the host process's working directory using the same `baml.toml` / `baml_src/` markers as the CLI. Use an absolute `BAML_HOME` when launching processes from different directories; a relative value is resolved against each process's working directory. An unwritable store disables recording with a diagnostic rather than silently writing into another store.
+
+Use `BOUNDARY_API_KEY=local` to select local recording and query even with a saved Boundary login; `baml query --local` selects local reads without changing how your application records. The fixed location does not enable recording that the runtime telemetry policy has disabled. SDK hosts launched outside a project and packed executables use `<BAML_HOME>/btel` directly. Packed executables do not adopt a project's store when launched inside it.
+
+For example, launch your computer-use demo from its project and query its recordings with the same environment. CLI entrypoints use this store too (`baml run main` below assumes the project defines `main`):
+
+```sh
+cd /path/to/demo-computer-use
+export BOUNDARY_API_KEY=local
+export BAML_TELEMETRY=high
+# Run your application using baml_bridge from this project or a child directory.
+baml run main
+baml query --local 'SELECT COUNT(*) FROM processes'
+```
+
+`baml query --from` also accepts an existing state directory, a directory containing `recordings/` and `cas/`, or the `recordings/` directory itself. A recognized traces directory selects local querying directly, even with cloud credentials set. It takes precedence over project discovery and `BAML_HOME`; it cannot be combined with cloud `--project` or `--environment` selectors. An explicit project root uses its fixed project store even if it contains unrelated `btel/` or `recordings/` directories. These commands all read the same store:
+
+```sh
+# Replace the example suffix with the hash of your project's canonical path.
+store="${BAML_HOME:-$HOME/.baml}/demo-computer-use-<path-hash>"
+baml query --from "$store" 'SELECT COUNT(*) FROM processes'
+baml query --from "$store/btel" 'SELECT COUNT(*) FROM processes'
+baml query --from "$store/btel/recordings" 'SELECT COUNT(*) FROM processes'
+# Query recordings from packed programs or SDK hosts without a project:
+baml query --from "${BAML_HOME:-$HOME/.baml}" 'SELECT COUNT(*) FROM processes'
+```
+
+Compatibility: existing recordings are not moved or combined automatically. Older `<project-root>/.baml` stores and formerly configured stores remain readable by passing their state directory explicitly to `baml query --from`. `[tracing].store` no longer selects a destination; remove the obsolete table from `baml.toml`.
+
+### Querying large stores
+
+The first query builds an index of the selected store; later refreshes reconcile completed files using metadata and decode new or changed files. Refresh still lists every recording and stats completed files, so a global store accumulating many projects has more discovery and reconciliation work than a project store. SQL `LIMIT` and `--max-wall` constrain query execution, not this preceding refresh. `--format json` exposes `outcome.refresh` counters/timings and `outcome.query` timings to distinguish those costs.
+
+The default per-project stores limit how much unrelated history each refresh visits. After building the index, `baml query --from /path/to/store --no-refresh 'SELECT COUNT(*) FROM processes'` skips file reconciliation and queries the existing index; it does not include recordings added since the last refresh. Queries can still scan indexed rows and report extents for all recordings, and captured-value expressions can read CAS blobs. This path/configuration change does not claim a speedup for SQL over the same global dataset; global-store query performance is being investigated separately.

baml_language/crates/baml_cli/Cargo.toml

@@ -86,7 +86,9 @@ internal = [  ]
 skip-integ-tests = [  ]
 
 [dev-dependencies]
+baml_test_support = { workspace = true }
 baml_tests = { workspace = true }
+bridge_cffi = { workspace = true }
 btel_bcs = { workspace = true }
 btel_file = { workspace = true }
 btel_recorder = { workspace = true }

baml_language/crates/baml_cli/src/query_command.rs

@@ -1,6 +1,6 @@
-//! `baml query`: SQL over the project's local Btel recordings.
+//! `baml query`: SQL over local Btel recordings or the Boundary cloud.
 //!
-//! The first query creates `<project>/.baml/btel/query.sqlite`; each query
+//! The first local query creates `query.sqlite` in the selected Btel root; each query
 //! then applies only newly completed recording files before running. Rows
 //! and a terminal outcome (indexed extents, evidence diagnostics) are
 //! printed; JSON keeps diagnostics out of row data.
@@ -12,7 +12,9 @@
 
 use std::{io::Read as _, path::PathBuf, time::Duration};
 
-use baml_query_btel::{Budgets, Error, Index, IndexOptions, QueryRequest, Status, catalog, format};
+use baml_query_btel::{
+    Budgets, Error, Index, IndexOptions, QueryRequest, SourceLayout, Status, catalog, format,
+};
 use clap::{Parser, ValueEnum};
 
 use crate::project_load::find_project_root_from;
@@ -56,7 +58,9 @@ pub struct QueryArgs {
     #[arg(long, conflicts_with = "local")]
     pub environment: Option<String>,
 
-    /// Project directory (defaults to the current directory's project).
+    /// Project or traces directory: a project store under `BAML_HOME`, a legacy `.baml`,
+    /// btel directory, or recordings directory. A traces directory selects local
+    /// querying directly. Defaults to the shared local store for this project.
     #[arg(long, value_name = "PATH")]
     pub from: Option<PathBuf>,
 
@@ -165,6 +169,7 @@ impl QueryArgs {
         crate::cloud_query::run(self, endpoint, credential, source, project, sql)
     }
 
+    /// Select a source, refresh local files when requested, and render rows and outcomes.
     pub fn run(&self) -> anyhow::Result<crate::ExitCode> {
         if self.schema {
             return self.print_schema();
@@ -178,14 +183,38 @@ impl QueryArgs {
             Some(sql) => sql.to_owned(),
             None => anyhow::bail!("provide a SELECT statement, or --schema to list relations"),
         };
-        let root = match find_project_root_from(self.from.as_deref())? {
-            Some(root) => root,
+        // Trace folders beat enclosing projects and saved cloud credentials.
+        // An explicitly selected project root still uses its own configuration,
+        // even if it contains unrelated directories named btel or recordings.
+        let from = self
+            .from
+            .as_deref()
+            .map(|from| {
+                std::fs::canonicalize(from)
+                    .map_err(|error| anyhow::anyhow!("--from {}: {error}", from.display()))
+            })
+            .transpose()?;
+        let project_root = find_project_root_from(from.as_deref())?;
+        let trace_layout = from
+            .as_deref()
+            .filter(|from| {
+                !from.join(baml_db::BAML_TOML).is_file()
+                    && !from.join(baml_db::BAML_SRC_DIR).is_dir()
+            })
+            .and_then(SourceLayout::from_trace_directory);
+        let local = self.local || trace_layout.is_some();
+        anyhow::ensure!(
+            trace_layout.is_none() || (self.project.is_none() && self.environment.is_none()),
+            "--from selects a local traces directory; use a project directory for a cloud query"
+        );
+        let root = match project_root.as_ref() {
+            Some(root) => root.clone(),
             None => match &self.from {
                 Some(from) => from.clone(),
                 None => std::env::current_dir()?,
             },
         };
-        let api_key = if self.local {
+        let api_key = if local {
             None
         } else {
             baml_env::string_var("BOUNDARY_API_KEY")?
@@ -195,7 +224,7 @@ impl QueryArgs {
             !local_from_env || (self.project.is_none() && self.environment.is_none()),
             "BOUNDARY_API_KEY=local selects local recordings; unset BOUNDARY_API_KEY to use --project or --environment for a cloud query"
         );
-        if !self.local && !local_from_env {
+        if !local && !local_from_env {
             let settings = crate::cloud_config::Boundary::read(&root)?;
             let endpoint = settings.endpoint().map_err(|error| {
                 bcs_api::diagnostics::Context {
@@ -259,7 +288,19 @@ impl QueryArgs {
             },
             explain: self.explain,
         };
-        let outcome = Index::for_project(&root, options).and_then(|mut index| {
+        let layout = match trace_layout {
+            Some(layout) => layout,
+            None => {
+                let store = btel_settings::local_store::resolve(
+                    project_root.as_deref().or(self.from.as_deref()),
+                    baml_env::os_var("BAML_HOME"),
+                    std::env::home_dir(),
+                )?;
+                SourceLayout::for_btel(store.join("btel"))
+            }
+        };
+        let recordings = layout.recordings.clone();
+        let outcome = Index::open(layout, options).and_then(|mut index| {
             if self.no_refresh {
                 index.query(&request)
             } else {
@@ -287,8 +328,8 @@ impl QueryArgs {
         }
         if result.outcome.source_missing && self.format == QueryFormat::Table {
             eprintln!(
-                "note: no recordings at {}; run BAML code in this project to record some",
-                root.join(".baml/btel/recordings").display()
+                "note: no recordings at {}; run BAML code with this local store to record some",
+                recordings.display()
             );
         }
         Ok(match result.outcome.status {

baml_language/crates/baml_cli/src/runtime_telemetry.rs

@@ -15,6 +15,7 @@ pub(crate) fn create_engine(
     create_engine_with_context(program, argv, project_root, sources, Default::default())
 }
 
+/// Create the CLI engine with its launch context and shared local-store fallback.
 pub(crate) fn create_engine_with_context(
     program: bex_vm_types::Program,
     argv: Vec<String>,
@@ -37,7 +38,7 @@ pub(crate) fn create_engine_with_context(
                     boundary.api_url.as_deref(),
                 )
                 .unwrap_or_else(|| {
-                    TelemetryRecording::local_files(
+                    TelemetryRecording::project_files(
                         project_root,
                         btel_settings::publisher::RecordingConfig::default(),
                     )

baml_language/crates/baml_cli/tests/boundary_telemetry_e2e.rs

@@ -373,13 +373,22 @@ async fn initial_connection_failures_warn_without_changing_execution() {
     );
 }
 
+/// The local sentinel must keep both recording and querying away from the cloud.
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
 async fn local_sentinel_records_and_queries_locally_without_boundary_requests() {
     let temp = fixture("function main() -> int { 7 }\n");
     let server = MockServer::start().await;
     let output = execute_case(temp.path(), &server.uri(), "local", &["run", "main"]);
     assert!(output.status.success(), "{output:?}");
-    assert!(temp.path().join(RECORDINGS).exists());
+    // Local writers and readers now honor this fixture's BAML_HOME.
+    let store = btel_settings::local_store::resolve(
+        Some(temp.path()),
+        Some(temp.path().join("home").into()),
+        None,
+    )
+    .unwrap();
+    assert!(store.join("btel/recordings").exists());
+    assert!(!temp.path().join(RECORDINGS).exists());
     let output = execute_case(
         temp.path(),
         &server.uri(),

baml_language/crates/baml_cli/tests/embedded_telemetry_e2e.rs

@@ -788,6 +788,7 @@ async fn wrapper_build_embeds_named_api_key_for_both_release_variants() {
     }
 }
 
+/// Packed wrapper recordings stay in the user store and keep boundary spans value-free.
 #[test]
 fn wrapper_command_boundaries_record_spans_without_values() {
     let built = common::ensure_built();
@@ -843,7 +844,7 @@ fn wrapper_command_boundaries_record_spans_without_values() {
     let query = |sql: &str| {
         let output = Command::new(&built.baml_cli)
             .args(["query", "--local", "--format", "json", "--from"])
-            .arg(&home)
+            .arg(home.join(".baml"))
             .arg(sql)
             .env("BAML_HOME", home.join(".baml"))
             .env("BAML_CLI_ALLOW_DIRECT", "1")

baml_language/crates/baml_cli/tests/query_e2e.rs

@@ -20,11 +20,15 @@ function main() -> int {
 }
 ";
 
+/// Run with isolated local state so saved credentials cannot change the data source.
 fn cli(project: &Path, args: &[&str], envs: &[(&str, &str)]) -> std::process::Output {
     Command::new(env!("CARGO_BIN_EXE_baml-cli"))
         .args(args)
         .current_dir(project)
         .env("DEV_BAML_CLI_DISABLE_AGENT_DETECTION", "1")
+        .env("BAML_HOME", project.join(".baml"))
+        .env("BOUNDARY_API_KEY", "local")
+        .env("DO_NOT_TRACK", "1")
         .env_remove("BAML_TELEMETRY")
         .envs(envs.iter().copied())
         .output()
@@ -204,6 +208,7 @@ fn fresh_query_processes_index_once_and_then_only_new_files() {
     assert_eq!(calls["outcome"]["status"], "complete");
 }
 
+/// Schema, SQL rejection, and stdin framing must also work with an empty local store.
 #[test]
 fn schema_invalid_sql_and_stdin() {
     let project = tempfile::tempdir().unwrap();
@@ -231,6 +236,9 @@ fn schema_invalid_sql_and_stdin() {
         .args(["query", "--local", "--format", "jsonl", "-"])
         .current_dir(project.path())
         .env("DEV_BAML_CLI_DISABLE_AGENT_DETECTION", "1")
+        .env("BAML_HOME", project.path().join(".baml"))
+        .env("BOUNDARY_API_KEY", "local")
+        .env("DO_NOT_TRACK", "1")
         .stdin(std::process::Stdio::piped())
         .stdout(std::process::Stdio::piped())
         .spawn()

baml_language/crates/baml_cli/tests/run_context_e2e.rs

@@ -11,6 +11,7 @@ use std::{
 
 const CONTEXT: &str = r#"{"distinct_id":"pipe-user","metadata":{"phase":"launch"}}"#;
 
+/// Keep context-query evidence isolated while sharing only the build-cache home.
 fn project() -> tempfile::TempDir {
     let project = tempfile::tempdir().unwrap();
     common::write_project(
@@ -51,6 +52,7 @@ fn piped_cli_with_env_context(
         .env("DEV_BAML_CLI_DISABLE_AGENT_DETECTION", "1")
         .env("BAML_CLI_ALLOW_DIRECT", "1")
         .env("BAML_TELEMETRY", "high")
+        .env("BOUNDARY_API_KEY", "local")
         .env("BAML_HOME", common::shared_baml_home())
         .stdin(Stdio::piped())
         .stdout(Stdio::piped())