Calc Flow

Calc Flow is a Python calculation library for Arrow tables and stateful streams. Compose typed expressions, SQL, and reusable Python functions in one pipeline. Collect a dataset or iterate its results as data arrives. Rust provides the internal runtime: DataFusion table execution, graph compilation, state, checkpoints, and recovery. Calc Flow Studio, the local FastAPI and React application, lives in the separate calc-flow-studio repository.

Install

CPython 3.9 or newer:

uv add calc-flow-python

Optional array providers:

uv add "calc-flow-python[numpy]"
uv add "calc-flow-python[jax]"

Python quickstart

import pyarrow as pa
import calc_flow as cf

data = pa.table({"a": [1, 3], "b": [2, 4]})
result = cf.compute(data, lambda t: t.select(total=t["a"] + t["b"]))
assert result.to_pydict() == {"total": [3, 7]}

cf.compute infers the supported Arrow schema and returns a pyarrow.Table. The builder receives a TableExpr; indexing selects columns, operators compose calculations, and select, with_columns, and filter return new declarations. Use await cf.compute_async(...) in an event loop.

Add a read-only SQL stage with t.sql(query), using the local alias input. Use cf.sql(query, orders=orders, fees=fees) for explicit named table declarations. SQL returns another TableExpr, so pipe, column operators, and table methods compose before and after it. expression.pipe(function, *args, **kwargs) calls an ordinary synchronous function once while building the calculation.

For streams, use async with output.stream(batches()) as results, then async for table in results. One native job retains state across batches and owns cleanup. Declared event-time inputs default to validated nondecreasing timestamps and watermarks that produce finalized results before EOF. Use watermarks for explicit disorder or source-provided progress; see watermark policies. A Program yields named StreamOutput events for multiple outputs. The convenience stream uses temporary checkpoints and best-effort iterable delivery; durable recovery uses explicit source/sink bindings and a managed checkpoint root.

Continue with SQL and reusable pipelines and the first streaming pipeline. The runnable examples cover SQL composition, stateful streaming, and named streaming outputs. The connector examples cover all seven transports and every supported read/write direction, with service setup and output checks. The default wheel includes file; other transports require native build features. Expression workflows cover financial features, recovery, static matrices, and joins. Use explicit Runtime, plans, Batch, UDFs, and PipelineBuilder for runtime integrations and diagnostics.

The Rust runtime reference and native examples cover implementation and extension work. The calc-flow crate's exports remain available to those users.

Architecture

crates/calc-flow  (Rust core: Batch, graph compiler, DataFusion, runners, stores)
  ├─ crates/calc-flow-connectors  (trusted transport implementations)
  └─ crates/calc-flow-python  (PyO3 _native binding + registered connectors)
       └─ python/calc_flow  (Python expressions, Arrow execution + integrations)

The native dependency edges are crates/calc-flow ← calc-flow-connectors and crates/calc-flow ← crates/calc-flow-python ← python/calc_flow. The Python package is not a second engine.

Path	Purpose
crates/calc-flow/	Native core: batches, ports/operators, graph compiler, DataFusion runtime, UDF/provider registries, runners, checkpoints, project stores
crates/calc-flow-connectors/	Trusted file, Kafka, PostgreSQL, MySQL, ClickHouse, HTTP, and WebSocket connectors behind feature gates
crates/calc-flow-python/	PyO3 binding exposing the core as calc_flow._native
python/calc_flow/	Python expressions and SQL, pipe, Arrow collection, owned stream results, lowering, and runtime integrations
schemas/	project-v3.schema.json, the canonical generated project contract
examples/	Executable Python expression and integration examples
benchmarks/	Benchmark workloads; scheduled regression gates and informational comparisons

Data and execution model

• Table data is Arrow-backed. The Rust runtime executes row expressions and SQL with DataFusion and owns native rolling, window, and cross-section operators.
• NumPy and JAX are optional Python array providers. They are registered explicitly and evaluate a bounded, allowlisted expression language.
• Raw tables or arrays never cross a graph or runner boundary; they are wrapped in immutable Batch envelopes.
• Project documents are strict, data-only JSON/YAML with format_version: 3. They select batch or stream runtime mode explicitly; stream documents reference registered connectors and named secrets without embedding credentials, callables, import paths, or table backend selectors.
• Table and mixed graph runs own one run-scoped DataFusion session. External-only NumPy/JAX runs own no DataFusion configuration, UDF state, or runtime and return an empty DataFusion metrics list.
• Convenience batch calls return Arrow tables. Explicit plan execution returns named Batch outputs, per-node timings, metadata, and DataFusion diagnostics.
• Python executions accept reusable frozen ExecutionOptions with deep-copied strict-JSON settings and a cooperative, timezone-aware deadline normalized to UTC.
• TableExpr.stream and Program.stream own a single native job with bounded backpressure. SQL accepts one alias in a stream. Ordinary aggregates and GROUP BY emit full cumulative snapshots; other SQL runs per batch.
• Program(name, engine="sql") or Program(name, engine="streaming") fixes the engine when the immutable program is created. Program.execute returns finite named tables or owned streaming results, respectively. Calls for the other engine fail.
• The source-driven StreamingRunner consumes a StreamExecutionPlan, owns async source/sink bindings, and returns a one-owner StreamingJob.
• Managed epoch checkpoints use LocalStateBackend segments and strict v3 CheckpointManifest documents. Exactly-once compatibility is proved per requested output; ordinary sinks can provide at-least-once delivery on a lossless replayable route. Async iterable convenience inputs provide best effort.

The capabilities and execution model are introduced in docs/introduction.md. The complete component and lifecycle design is in docs/design.md, and the practical continuous tutorial is docs/streaming-guide.md.

Trusted extensions

Python applications may register trusted vectorized DataFusion scalar UDFs on a Runtime. Every registration declares provider, name, version, exact Arrow input types, return type, and volatility. Graph nodes select registrations explicitly with (provider, name, version) references. Serialized projects contain references only.

Runtime extension authors use UdfRegistry for native DataFusion UDFs and ProviderRegistry for explicitly registered external operators.

Project contracts

Calc Flow accepts strict project-v3 documents. Read projects and persistence for validation, serialization, and reloading a graph. Historical changes are recorded in CHANGELOG.md.

Development

Large Cargo and Maturin outputs should use the repository target/ tree. The complete CI/full-verification command reference is below. Local changes use the smallest affected checks under AGENTS.md Verification; full regression and Rust 90% coverage belong to CI; routine benchmark measurements run on their own schedule:

uv sync --extra dev
cargo fmt --all --check
cargo clippy --workspace --lib --bins --tests --examples --all-features -- -D warnings
uv run python scripts/run_rust_tests.py
CALC_FLOW_CONNECTOR_CONTAINERS=1 \
  CALC_FLOW_KAFKA_BOOTSTRAP=localhost:9092 \
  CALC_FLOW_PG_TEST_URL=postgresql://postgres:postgres@localhost:5432/postgres \
  CALC_FLOW_MYSQL_TEST_URL=mysql://root:calcflow-test@localhost:3306/calcflow \
  CH_TEST_URL=http://localhost:8123 \
  uv run python scripts/run_rust_coverage.py
RUSTDOCFLAGS="-D warnings" cargo doc --workspace --all-features --no-deps

uv run maturin develop
JAX_PLATFORMS=cpu uv run pytest python/tests -q
JAX_PLATFORMS=cpu uv run python scripts/run_examples.py
uv run ruff check .
uv run ruff format --check .

Regular CI runs cargo audit, cargo deny --locked check, core package inspectors, and isolated wheel smoke tests. The Python release workflow builds, verifies, and publishes core package artifacts. See AGENTS.md for the maintained repository commands and constraints.

Documentation

• Documentation index — reading order for all published docs
• Introduction — capabilities, vocabulary, and execution modes
• getting started — installation and smoke test
• Executable examples — verified example matrix and runner
• Batch calculations — expressions, SQL, UDFs, and async execution
• Arrays and matrices — NumPy/JAX and static weights
• Continuous streaming — source-to-recovery tutorial
• Connectors — transport configuration and guarantees
• Projects and persistence — JSON/YAML and file stores
• Python API — Python surface and examples
• Expression workflows — batch, stream, recovery, static matrix, and performance workflows
• Rust runtime reference — native surface and examples
• API reference — supported surfaces at a glance
• Expression API — declarations, analysis, and compile requirements
• Design and architecture — component ownership and execution design
• Verification — documentation, examples, and implementation checks
• Python release guide — packaging, verification, Trusted Publishers, and the PyPI procedure
• Benchmark suite — scheduled tables, scale matrices, external-engine comparisons and historical regression evidence
• Changelog — the single history of changes

License

Apache-2.0 — see LICENSE.