ORCA: Observability with Realtime Control and Aggregation

Docs

Entry point: docs/Home.md

Tools

• libunwind_example: isolated POC for libunwind-based utilities
• dyninst_example: isolated POC for Dyninst-based utilities
• bpf_example: isolated POC for BPF-based utilities. Requires helper to be compiled using out-of-tree tools.
• rust-examples/iceberg-main: Iceberg POC converting ORCA-generated Parquet into Iceberg, usable for analysis with tools such as Polars afterward. Does not assume the presence of a catalog service; a temporary in-memory catalog is provisioned to facilitate conversion and discarded afterward.

Compiling bpf_helper:

clang -target bpf -O2 -g -c tools/examples/bpf_helper.bpf.c -o /tmp/bpf_helper.bpf.o

Known Issues

• Failure handling is very limited (AGGs transition to failure but still continue processing data)
• Overlay msgslot path is only used in some places
• msgslot and ratelimiter can probably be merged

Commands Supported

Core Commands

• pause: Pauses execution on all MPI ranks
• resume: Resumes execution on all MPI ranks. MPI tier bootstraps in paused mode. resume must be a part of the initial bootstrap command.

Flow Configuration (set-flow)

flow: current analytics 'spec' (flow plan). Each set-flow command overwrites the previous one.

set-flow file [yaml_path]:

• Read flow config from a YAML. If provided, yaml_path is used. Otherwise, the default flow config is used.
• Internally, this maps to ExecuteMode::kCompute.
• Tracers not referenced in the flow config are explicitly disabled to prevent data collection (even though the plan would anyway drop them later).

set-flow enable-tracers [stream_names...]:

• Enables only the specified streams and disables all others.
• Internally, MPI is set to kForward (passthrough), and AGG is set to kPersist.
• If no streams are provided, it enables all available tracers.

set-flow disable-tracers [stream_names...]:

• Disables the specified streams and enables all others.
• Internally, MPI is set to kForward (passthrough), and AGG is set to kPersist.
• If no streams are provided, all are disabled

set-flow trace-and-drop-mpi [stream_names...]:

• Data is collected at MPI level, but then dropped. Useful for testing/benchmarking.
• Internally, MPI is set to kDrop, and AGG does not matter.

set-flow trace-and-drop-agg [stream_names...]:

• Data is collected at AGG level, but then dropped. Useful for testing/benchmarking.
• Internally, MPI is set to kForward (passthrough), and AGG is set to kDrop.

Probe Control

Probes allow for granular control over intra-schema data collection. By default, all probes are enabled if a schema is enabled. This mechanism does not decide whether a schema is enabled or disabled, it only decides what would be collected if the schema were enabled.

• enable-probe <schema_name> <probe_name>: Enables a specific probe within a schema.
• disable-probe <schema_name> <probe_name>: Disables a specific probe within a schema. When a probe is disabled, its AddRow calls in the application are short-circuited, incurring near-zero overhead.

---

License

ORCA is released under the MIT License. See LICENSE, except where otherwise noted.

The src/ingest/duckdb/ FlightSQL implementation includes code adapted from VoltronData sqlflite and is licensed under Apache License 2.0. See src/ingest/duckdb/README.md.

This software was developed, in part, under US government contract 89233218CNA000001, for Los Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC for the US Department of Energy/National Nuclear Security Administration.