Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,61 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]


## [0.10.0] - 2026-08-07

### Added

- `create_index(database, table, columns=..., index_type=...)` builds an index on a
managed table, bringing the framework client to parity with `hotdata indexes
create` in the CLI. It covers all three index kinds the API accepts: `"bm25"` for
full-text search, `"vector"` for nearest-neighbour search, and `"sorted"`.
Previously the framework had no index API at all and callers had to drop to the raw
`hotdata.IndexesApi`, which left a managed database's data loaded but not
searchable: full-text queries error without an index, and vector queries run at
full-scan speed. Like the other managed-table operations, `database` accepts a
name/id or an already-resolved `ManagedDatabase`. Indexing a table on a plain
(non-managed) connection is not covered — the CLI's `--catalog` handles that.

`index_name` is optional and defaults to `{table}_{columns}_{index_type}`, the same
derivation the CLI uses when `--name` is omitted, so both surfaces name the same
index identically. `index_type` is required, unlike the API's `"sorted"` default,
because the wrong kind only fails at query time.

The server builds the index as a background job whose submit call reports success
even when the build later fails, so `create_index` polls the job to a terminal
state and raises `RuntimeError` carrying the job's `error_message`. Pass
`wait=False` to return once the job is accepted (`status="pending"` plus a
`job_id`) and own the outcome check yourself, as the CLI's `--async` does;
`timeout_s` and `poll_interval_s` tune the wait.

Both vector-index modes are supported. Omitting `embedding_provider_id` indexes an
existing vector column, queried with a literal vector — and there `metric` must
match the distance function the query uses (`cosine`→`cosine_distance`,
`l2`→`l2_distance`, `dot`→`negative_dot_product`), since a mismatch silently
reverts to a full table scan rather than erroring. Setting
`embedding_provider_id` indexes a *text* column instead: the provider embeds it
into `output_column`, queries pass text via `vector_distance(source_col, 'query')`,
and the server resolves the distance function itself.

Argument combinations that the server would silently ignore raise `ValueError`
before any request is sent: an unknown `index_type` or `metric`, a vector index
with more than one column (the engine indexes only the first), and
`metric`/`dimensions`/`embedding_provider_id`/`output_column`/`description` on a
non-vector index.

Verified against `api.hotdata.dev` when this version was released: BM25 and vector
indexes both build and report `ready`, and a BM25 index is used by full-text
search. A *vector* index on a managed database was **not** picked up by the query
planner at that time — a matching `cosine_distance(...) ORDER BY ... LIMIT k` still
planned as a full scan. That reproduces with an index created by `hotdata indexes
create`, so it is an engine-side issue rather than a client one, but it means a
vector index built through this method may not yet accelerate queries.

- `CreateIndexResult`, the frozen dataclass `create_index` returns, is exported from
`hotdata_framework` and added to the public contract surface. Its `source_column`
names the text column to query for a provider-backed vector index, and is `None`
for BM25, sorted, and plain vector indexes.

## [0.9.0] - 2026-07-23

### Added
Expand Down
4 changes: 3 additions & 1 deletion CONTRACT.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ The supported import surface is:
- `ManagedDatabase`
- `ManagedTable`
- `LoadManagedTableResult`
- `CreateIndexResult`
- `DEFAULT_SCHEMA`
- `is_parquet_path`

Expand Down Expand Up @@ -63,7 +64,8 @@ Adapters should import from `hotdata_framework` and treat this surface as the st
- `upload_parquet(path)` uploads a local parquet file and returns an upload id.
- `load_managed_table(database, table, schema=..., upload_id=..., file=...)` publishes parquet data into a declared managed table.
- `delete_managed_table(database, table, schema=...)` deletes a managed table.
- The `database` argument of `list_managed_tables`, `load_managed_table`, `add_managed_table`, `delete_managed_table`, `delete_managed_database`, and `execute_sql` accepts a name/id **or** an already-resolved `ManagedDatabase`. Passing a `ManagedDatabase` skips the name/id read probe, so a create-scoped key that cannot read `/databases` can load into a database it just created.
- `create_index(database, table, schema=..., columns=..., index_type=..., index_name=...)` builds a `"sorted"`, `"bm25"`, or `"vector"` index on a managed table and returns a `CreateIndexResult`. It is the framework-side equivalent of the CLI's `hotdata indexes create`; indexing a table on a plain (non-managed) connection is out of scope. `index_name` defaults to `{table}_{columns}_{index_type}`, matching the CLI's derivation when `--name` is omitted. `index_type` is required rather than defaulting to the API's `"sorted"`. The build runs as a background job; the call polls it to a terminal state and raises `RuntimeError` with the job's `error_message` when it fails, because the submit call reports success regardless. `wait=False` returns as soon as the job is accepted, with `status="pending"` and a `job_id` for the caller to poll. For `index_type="vector"`, omitting `embedding_provider_id` indexes an existing vector column and `metric` (`"l2"`, `"cosine"`, `"dot"`) selects the distance function the index accelerates — a query using a different function silently falls back to a full scan; setting `embedding_provider_id` indexes a source *text* column instead, and the returned `source_column` names the column to pass to `vector_distance`. Argument combinations the server would silently ignore raise `ValueError` before any request is sent.
- The `database` argument of `list_managed_tables`, `load_managed_table`, `add_managed_table`, `delete_managed_table`, `delete_managed_database`, `create_index`, and `execute_sql` accepts a name/id **or** an already-resolved `ManagedDatabase`. Passing a `ManagedDatabase` skips the name/id read probe, so a create-scoped key that cannot read `/databases` can load into a database it just created.

### `QueryResult`

Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ Runtime boundary and guarantees are defined in `CONTRACT.md`.
- **Result utilities** — convert query results to records, pandas DataFrames, or metadata dictionaries for adapter display layers.
- **History helpers** — list recent results and query run history with normalized dataclasses.
- **Managed databases** — create Hotdata-owned catalogs, declare tables, upload parquet, and load managed tables (mirrors `hotdata databases` in the CLI).
- **Indexes** — build BM25, vector, or sorted indexes on managed tables, mirroring `hotdata indexes create` (managed databases only). Waits on the background build job and surfaces its failure, instead of reporting the phantom success the submit call returns.
- **Health helpers** — build compact API/workspace health summaries for UI integrations.

Install:
Expand Down
2 changes: 2 additions & 0 deletions hotdata_framework/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
)
from hotdata_framework.databases import (
DEFAULT_SCHEMA,
CreateIndexResult,
LoadManagedTableResult,
ManagedDatabase,
ManagedTable,
Expand Down Expand Up @@ -43,6 +44,7 @@

__all__ = [
"DEFAULT_SCHEMA",
"CreateIndexResult",
"HotdataClient",
"HotdataError",
"HotdataTerminalError",
Expand Down
Loading
Loading