Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,13 +136,13 @@ dialect or backend you use:
| Local/S3 Iceberg and Lance | Ray Data / Daft public readers | `tributo[data,data-daft]` | Alpha; real dual-engine Conformance |
| PostgreSQL structured table | Ray Data / Daft SQL readers | `tributo[postgresql,data-daft]` | Alpha; real PostgreSQL Conformance |
| HDFS Parquet/CSV | Ray Data + PyArrow Hadoop filesystem | Ray runtime with HDFS libraries | Adapter present; cluster gate pending |
| ClickHouse | `ray-clickhouse==0.1.0` / `daft-clickhouse==1.0` | `tributo[clickhouse]` plus the external Ray wheel when selecting Ray | Adapter only; the full image contains the Daft package, while the Ray package uses the external wheelhouse until PyPI publication and real-database Conformance remains the support gate |
| ClickHouse | `ray-clickhouse==0.1.0` external wheel | `tributo[clickhouse]` plus the external Ray wheel | Adapter only; the Ray package uses the external wheelhouse until PyPI publication and real-database Conformance remains the support gate |
| Doris | `ray-doris==1.0` / `daft-doris==1.0` | `tributo[mysql]` or `tributo[doris-flight]` | Adapter only; Ray routes require `ray-doris`, Daft routes require `daft-doris`, and real-database Conformance remains the support gate |
| HiveServer2 structured tables | `ray-hive==1.0` through the built-in Ray Binding | `tributo[hive-ray]` or `uv sync --locked --no-dev --extra hive-ray` | Alpha; real Hive 4.2.0 structured-projection Conformance |
| Native ORC files | no built-in reader | — | Unsupported; use an engine or table service that owns ORC decoding |

Provider/binding presence is not a support claim. The canonical full image
contains the locked v1.0 Daft ClickHouse and Doris connector packages, but the
contains the locked v1.0 Doris connector packages, but the
Ray ClickHouse route requires the external `ray-clickhouse==0.1.0` wheel until
that package is published to PyPI. These paths remain adapter-only until their
real-database Conformance gates pass.
Expand Down
2 changes: 1 addition & 1 deletion docs/STABILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ covered by the external `runtime-image` suite.
| Surface | Level | Notes |
|--------|-------|-------|
| `tools/build_tributo_image.py` | `alpha` | JSON-only Buildx builder for the pinned image validated for CPU execution by default, with explicit `linux/amd64`/`linux/arm64` targeting; it performs dependency-closure discovery, manifest sealing, and fail-closed import checks |
| `tools/tributo-runtime-full.json` | `alpha` | Multi-architecture pinned Ray/uv image references, native-platform default, complete first-party runtime-extra closure including locked v1.0 Daft ClickHouse/Doris and Ray Doris connectors plus `ray-hive`; optional external wheelhouse support supplies `ray-clickhouse==0.1.0` until PyPI publication |
| `tools/tributo-runtime-full.json` | `alpha` | Multi-architecture pinned Ray/uv image references, native-platform default, complete first-party runtime-extra closure including locked v1.0 Daft Doris and Ray Doris connectors plus `ray-hive`; optional external wheelhouse support supplies `ray-clickhouse==0.1.0` until PyPI publication |
| `manifest.json` / `image-profile.json` | `alpha` | Build attestations consumed for immutable image selection and algorithm artifact compatibility; not a registry or deployment API |

### Training (tributo.training.*)
Expand Down
6 changes: 3 additions & 3 deletions docs/architecture/call-chain-inventory.md
Original file line number Diff line number Diff line change
Expand Up @@ -301,9 +301,9 @@ WriteBinding selection; native dependency import occurs at factory/execute
```

Selected optional integrations (`ray-clickhouse==0.1.0`, `ray-doris==1.0`,
`daft-doris==1.0`, `daft-clickhouse==1.0`) also have thin built-in descriptors
and explicit install diagnostics. The canonical full runtime locks the v1.0
packages into the image; `ray-clickhouse` remains an external-wheelhouse input
`daft-doris==1.0`) also have thin built-in descriptors and explicit install
diagnostics. The canonical full runtime locks the v1.0 Doris packages into the
image; `ray-clickhouse` remains an external-wheelhouse input
until PyPI publication. These adapters are not support claims until database
infrastructure gates pass. Ray and Daft routes remain explicit and are not
interchangeable.
Expand Down
10 changes: 5 additions & 5 deletions docs/data/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,18 +56,18 @@ selects `binding_id` explicitly.
| Local/S3 Lance | Native reader | Native reader | Verified |
| PostgreSQL structured table | Native SQL reader | Native SQL reader | Verified |
| HDFS Parquet/CSV | Native reader with PyArrow HDFS | No locked public reader | Adapted; cluster gate pending |
| ClickHouse | `ray-clickhouse==0.1.0` external wheel | `daft-clickhouse==1.0` | Adapter only; Ray uses the installed wheel through `ray_clickhouse.ray.clickhouse`, Daft uses `tributo[clickhouse]`, and real-database Conformance remains the support gate |
| ClickHouse | `ray-clickhouse==0.1.0` external wheel | No built-in route | Adapter only; Ray uses the installed wheel through `ray_clickhouse.ray.clickhouse`, and real-database Conformance remains the support gate |
| Doris | `ray-doris==1.0` | `daft-doris==1.0` | Adapter only; Ray routes use `ray-doris`, Daft routes use `daft-doris`, and the full runtime image contains both v1.0 packages |
| HiveServer2 structured table | `ray-hive==1.0` through `tributo.ray.hive` | No built-in route | Alpha, verified with Hive 4.2.0; projection and worker execution only through binary HiveServer2 |
| Native ORC file | No built-in route | No built-in route | Unsupported; the HiveServer2 path does not expose ORC/HDFS files |

“Verified” means the current combination has semantic Conformance and real
storage or database evidence. It does not turn every engine/source combination
into a supported path. See the [support matrix](../reference/support-matrix.md)
for the exact boundary. Ray ClickHouse, Daft ClickHouse, Daft Doris, and Ray
Doris are explicit engine routes; a missing wheel never causes an engine
fallback. Hive requests must explicitly select the Ray engine; Tributo does not
add a Daft route or fall back to file-level ORC/HDFS access.
for the exact boundary. Ray ClickHouse, Daft Doris, and Ray Doris are explicit
engine routes; a missing wheel never causes an engine fallback. Hive requests
must explicitly select the Ray engine; Tributo does not add a Daft route or fall
back to file-level ORC/HDFS access.

Credentials belong to runtime configuration. They must not appear in dataset
identifiers, logical plans, receipts, logs, or public errors. Bounded input
Expand Down
9 changes: 4 additions & 5 deletions docs/getting-started/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ Pydantic, ONNX Runtime, PyArrow, pandas, and S3 filesystem support.
| Daft ingestion | `uv sync --locked --no-dev --extra data --extra data-daft` |
| HiveServer2 via Ray Data connector package | `uv sync --locked --no-dev --extra hive-ray` |
| PostgreSQL ingestion | `uv sync --locked --no-dev --extra postgresql` |
| ClickHouse via Daft | `uv sync --locked --no-dev --extra clickhouse` |
| ClickHouse via Ray Data | Sync `clickhouse`, then install the external `ray-clickhouse==0.1.0` wheel into `.venv` |
| Doris via Daft/Ray Data | `uv sync --locked --no-dev --extra mysql` |
| Doris Flight via Daft/Ray Data | `uv sync --locked --no-dev --extra doris-flight` |
Expand All @@ -50,10 +49,10 @@ An extra installs dependencies. It does not turn a protocol, adapter, or
reserved problem type into a verified implementation. Check the
[support matrix](../reference/support-matrix.md) before deployment. Ray Tune
itself is included by the core Ray dependency; the `tune` extra adds the
optional BayesOpt search implementation. The `clickhouse` extra installs
`daft-clickhouse==1.0` and the shared ClickHouse driver dependencies. The Ray
route additionally requires the `ray-clickhouse==0.1.0` GitHub Release wheel
until its PyPI publication. `mysql` installs `daft-doris==1.0` and
optional BayesOpt search implementation. The `clickhouse` extra installs the
shared ClickHouse driver dependency. ClickHouse reads use the Ray route, which
additionally requires the `ray-clickhouse==0.1.0` GitHub Release wheel until its
PyPI publication. `mysql` installs `daft-doris==1.0` and
`ray-doris==1.0` for their explicit engine routes, while `doris-flight` adds
their Flight dependencies. The `hive-ray` extra installs `ray-hive==1.0` for
the built-in Ray-only HiveServer2 Provider/Binding route. This does not add
Expand Down
7 changes: 3 additions & 4 deletions docs/how-to/runtime-images.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,10 +60,9 @@ The output directory contains:
- `build.log`: the command/evidence log for this build.

The builder never pushes an image. Publishing requires a separate reviewed
workflow. The canonical configuration resolves `daft-clickhouse==1.0`,
`daft-doris==1.0`, `ray-doris==1.0`, and `ray-hive==1.0` through the locked
Tributo extras, so the normal image build does not require a local connector
wheelhouse. The
workflow. The canonical configuration resolves `daft-doris==1.0`,
`ray-doris==1.0`, and `ray-hive==1.0` through the locked Tributo extras, so the
normal image build does not require a local connector wheelhouse. The
`external_wheelhouse` option remains available for packages outside the lock:
it is copied into a named build context and installed with
`pip --no-index --no-deps`; every wheel is recorded by filename,
Expand Down
4 changes: 0 additions & 4 deletions docs/reference/api/data.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,6 @@ documentation for every public stability tier.

## `tributo.data.bindings._daft_sql`

```{autoclass} tributo.data.bindings._daft_sql.DaftClickHouseBinding
:no-members:
```

```{autoclass} tributo.data.bindings._daft_sql.DaftDorisBinding
:no-members:
```
Expand Down
4 changes: 2 additions & 2 deletions docs/reference/support-matrix.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ endpoint.
| PostgreSQL structured table reads | Verified | Ray uses a single public SQL read and fails closed on parallel shard requirements; Daft may use native partition hints |
| ClickHouse/Doris raw SQL | Unsupported | Legacy shapes return a credential-free migration error; use structured table input or execute SQL outside Tributo ingestion |
| HDFS Parquet/CSV reads | Adapter only | Ray binding exists; real HDFS/JVM/worker gate is pending |
| ClickHouse reads | Adapter only | Daft uses locked `daft-clickhouse==1.0`; Ray uses the separately installed `ray-clickhouse==0.1.0` wheel and delegates single, physical-partition, integer-range, and bounded Arrow block behavior to it. Real-database Tributo Conformance is still required |
| ClickHouse reads | Adapter only | Ray uses the separately installed `ray-clickhouse==0.1.0` wheel and delegates single, physical-partition, integer-range, and bounded Arrow block behavior to it. Real-database Tributo Conformance is still required |
| Doris reads | Adapter only | Ray routes use locked `ray-doris==1.0`; Daft routes use locked `daft-doris==1.0`; real-database Conformance is still required and tablet planning remains provider/binding-owned |
| HiveServer2 structured table reads | Alpha, verified | `ProviderSourceConfig(provider="tributo.hive", uri="hive://host:port/database/table")` routes only to the Ray `tributo.ray.hive` Binding and locked `ray-hive==1.0`; Hive 4.2.0 Conformance verifies projection, schema, worker execution, and binary NOSASL transport. Raw SQL, Daft Hive, HTTP, TLS, Kerberos, and Hive writes are outside this contract |
| Native ORC file reads | Unsupported | Tributo does not implement an ORC reader. The verified Hive fixture is ORC-backed, but HiveServer2 and `ray-hive` own its SQL execution and decoding; this is not HDFS or file-level ORC support |
Expand Down Expand Up @@ -173,7 +173,7 @@ tensors; it does not apply DNN/PU preprocessing implicitly.
| Full Tributo runtime image for CPU validation | Alpha, directly buildable | Linux `arm64`/`amd64`, defaulting to the native host architecture; Python 3.12, Ray 2.55.1, locked dependency closure, and all first-party runtime extras including Alpha modules. Linux PyTorch resolution may include transitive CUDA/NVIDIA distributions, but no GPU support is claimed |
| Custom connector wheelhouse variant | Alpha, optional extension | An external wheelhouse remains available for packages outside the locked connector set and is currently required for `ray-clickhouse==0.1.0`; it does not change the Tributo lockfile |
| Runtime image attestation | Alpha | `manifest.json`, `image-profile.json`, normalized distribution inventory, and sealed `org.tributo.manifest-sha256` label |
| Runtime image Ray Jobs gate | Alpha, validation gate | Requires a unique two-node Docker Ray cluster on a native host architecture; verifies driver/worker imports, Ray Data, Jobs API submission, and v1.0 ClickHouse/Doris/Ray Hive package presence. `linux/amd64` requires a matching native host |
| Runtime image Ray Jobs gate | Alpha, validation gate | Requires a unique two-node Docker Ray cluster on a native host architecture; verifies driver/worker imports, Ray Data, Jobs API submission, and the ClickHouse driver plus v1.0 Doris/Ray Hive connector package presence. `linux/amd64` requires a matching native host |
| GPU runtime image | Not implemented | The Linux dependency closure may contain transitive CUDA/NVIDIA distributions from PyTorch, but no GPU driver, scheduling, NCCL, or GPU compatibility contract has been validated |
| HiveServer2 Runtime | Alpha, validation gate | `ray-hive==1.0` is locked into the full image and selected by the built-in Ray Hive Binding; the Data Ingestion Gate verifies Hive 4.2.0 on a worker. HDFS and native ORC readers remain outside this runtime contract |

Expand Down
1 change: 0 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,6 @@ postgresql = [
]
clickhouse = [
"clickhouse-connect[arrow,async]>=1.5,<1.6",
"daft-clickhouse[clickhouse]==1.0",
]
mysql = [
"PyMySQL>=1.2,<1.3",
Expand Down
41 changes: 0 additions & 41 deletions src/tributo/data/bindings/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,6 @@
_DAFT_INSTALL_HINT = "pip install 'tributo[data-daft]'"
_DAFT_LANCE_INSTALL_HINT = "pip install 'tributo[data,data-daft]'"
_DATA_INSTALL_HINT = "pip install 'tributo[data]'"
_DAFT_CLICKHOUSE_INSTALL_HINT = (
"Install daft-clickhouse==1.0 with Tributo: "
"pip install 'tributo[clickhouse]' (or uv sync --extra clickhouse)"
)
_RAY_CLICKHOUSE_INSTALL_HINT = (
"Install the ray-clickhouse==0.1.0 wheel, then install Tributo's "
"ClickHouse dependencies with pip install 'tributo[clickhouse]'"
Expand Down Expand Up @@ -281,29 +277,6 @@ def _daft_lance_descriptor() -> BindingDescriptor:
)


def _daft_clickhouse_descriptor() -> BindingDescriptor:
from tributo.data.bindings.daft_clickhouse import DaftClickHouseBinding

return BindingDescriptor(
key=BindingKey(
"tributo.daft",
ScanKind.SQL,
"clickhouse",
"daft_clickhouse.daft.clickhouse",
),
factory=DaftClickHouseBinding,
capabilities=frozenset({SourceCapability.PROJECTION}),
distribution_name="daft-clickhouse",
distribution_version=_distribution_version("daft-clickhouse") or "1.0",
engine_version_spec=_DAFT_SQL_VERSION_SPEC,
dependency_distributions=("clickhouse-connect",),
supported_read_hints=frozenset(
{ReadHint.TARGET_PARALLELISM, ReadHint.BATCH_SIZE}
),
install_hint=_DAFT_CLICKHOUSE_INSTALL_HINT,
)


def _ray_clickhouse_descriptor() -> BindingDescriptor:
from tributo.data.bindings.ray_clickhouse import RayClickHouseBinding

Expand Down Expand Up @@ -673,20 +646,6 @@ def default_engine_bindings() -> EngineBindings:
None,
("pylance", "daft-lance"),
),
(
_daft_clickhouse_descriptor,
BindingKey(
"tributo.daft",
ScanKind.SQL,
"clickhouse",
"daft_clickhouse.daft.clickhouse",
),
"daft",
_DAFT_SQL_VERSION_SPEC,
_DAFT_CLICKHOUSE_INSTALL_HINT,
None,
("daft-clickhouse", "clickhouse-connect"),
),
(
_ray_clickhouse_descriptor,
BindingKey(
Expand Down
57 changes: 21 additions & 36 deletions src/tributo/data/bindings/_daft_sql.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Shared Daft SQL bindings for independent database connector packages."""
"""Daft SQL binding for the independent daft-doris connector package."""

from __future__ import annotations

Expand Down Expand Up @@ -75,15 +75,14 @@ def compile(self, request: BindingCompileRequest) -> BindingCompilation:
"target_parallelism"
),
)
if self.connector_id == "doris":
protocol = str(request.runtime_options.get("protocol") or "mysql")
if protocol not in {"mysql", "flight"}:
raise BindingStageError.framework_diagnostic(
"validate_capabilities",
error_type=JobConfigurationError,
diagnostic_code="unsupported_doris_transport",
diagnostic=("Doris transport must be 'mysql' or 'flight'"),
)
protocol = str(request.runtime_options.get("protocol") or "mysql")
if protocol not in {"mysql", "flight"}:
raise BindingStageError.framework_diagnostic(
"validate_capabilities",
error_type=JobConfigurationError,
diagnostic_code="unsupported_doris_transport",
diagnostic=("Doris transport must be 'mysql' or 'flight'"),
)
with binding_stage("classify_transforms"):
decisions = residual_decisions(request.transforms)
with binding_stage("build_native_plan"):
Expand Down Expand Up @@ -114,26 +113,19 @@ def _build(
if target_tasks is not None:
options["target_tasks"] = target_tasks

if self.connector_id == "clickhouse":
from daft_clickhouse import read_clickhouse
from daft_doris import read_doris

options["port"] = target.port
dataframe = read_clickhouse(**options)
transport_id = "clickhouse_native"
else:
from daft_doris import read_doris

protocol = str(request.runtime_options.get("protocol") or "mysql")
options["transport"] = protocol
options["mysql_port"] = target.port
http_port = request.runtime_options.get("http_port")
flight_port = request.runtime_options.get("flight_port")
if http_port is not None:
options["http_port"] = int(http_port)
if flight_port is not None:
options["flight_port"] = int(flight_port)
dataframe = read_doris(**options)
transport_id = protocol
protocol = str(request.runtime_options.get("protocol") or "mysql")
options["transport"] = protocol
options["mysql_port"] = target.port
http_port = request.runtime_options.get("http_port")
flight_port = request.runtime_options.get("flight_port")
if http_port is not None:
options["http_port"] = int(http_port)
if flight_port is not None:
options["flight_port"] = int(flight_port)
dataframe = read_doris(**options)
transport_id = protocol

schema = canonical_engine_schema(dataframe.schema())
transforms = ConcreteTransformCompiler().compile(
Expand Down Expand Up @@ -178,13 +170,6 @@ def _wrap(
)


@PublicAPI(stability=Stability.ALPHA)
class DaftClickHouseBinding(_DaftSqlBinding):
connector_id = "clickhouse"
package_name = "daft-clickhouse"
reader_api = "daft_clickhouse.read_clickhouse"


@PublicAPI(stability=Stability.ALPHA)
class DaftDorisBinding(_DaftSqlBinding):
connector_id = "doris"
Expand Down
5 changes: 0 additions & 5 deletions src/tributo/data/bindings/daft_clickhouse.py

This file was deleted.

Loading
Loading