feat: add create_index to HotdataClient

[rohan-hotdata]

Why

HotdataClient had no index API at all, so a caller could create a managed database and load data into it, then not make that data searchable — full-text queries error outright without an index, and vector queries run at full-scan speed. The only workaround was dropping to the raw hotdata.IndexesApi, which hotdata-langchain's BM25 demo does today.

This is a parity gap, not a new capability: hotdata indexes create has done this from the CLI all along. The CLI's flags are treated as the spec here.

Closes item 2 of hotdata-dev/hotdata-langchain#36.

What

create_index(database, table, *, columns, index_type, ...) -> CreateIndexResult, covering all three kinds the API accepts (bm25, vector, sorted) and both vector modes:

Mode	embedding_provider_id	columns is	Queried with
Plain	omitted	the vector column	cosine_distance(col, ARRAY[…])
Provider-backed	set	the source text column	vector_distance(col, 'text')

The returned source_column names the column to query in provider-backed mode, where the indexed column is the generated embedding column rather than the one you passed.

Design decisions worth reviewing

It polls the build job. With async, the create call returns SubmitJobResponse(status="pending") and looks successful even for a build that later fails — the failure appears only on JobsApi.get_job(id).error_message. Without polling, callers believe they have an index they don't have. wait=False opts out (the CLI's --async).

The generated client types create_index as returning IndexInfoResponse, but at runtime it returns that on 201 or SubmitJobResponse on 202, so the code dispatches on isinstance — the same shape _execute_sql_once already uses for QueryApi.query. (The Rust CLI hit this too and bypasses its typed SDK with a raw POST.)

Stricter than the API where the API fails silently. Each of these otherwise surfaces only as a slow query, never an error:

• index_type is required; the API would default it to "sorted".
• A vector index takes exactly one column — the engine indexes only columns[0] and drops the rest.
• metric / dimensions / embedding_provider_id / output_column / description are rejected on non-vector indexes. The server silently drops most of these.
• metric is validated against l2 / cosine / dot. Note the wire value is dot, not ip — ip is the internal USearch MetricKind name. Confirmed in runtimedb's metric_to_distance_function.

index_name is optional, deriving {table}_{columns}_{index_type} — byte-identical to the CLI's derivation when --name is omitted, so both surfaces name the same index the same way.

Scope: managed databases only. The CLI's --catalog also accepts a plain connection; that's deliberately out of scope here, consistent with every other managed-table method on this client, and stated in the docstring.

Verification

Contracts checked statically against hotdata 0.8.0 and the runtimedb / hotdata-cli sources: same endpoint path as the CLI, same request body shape, var_async really serializes to the "async" wire alias, and managed-DB resolution to default_connection_id matches try_resolve_connection_id.

Then run end to end against api.hotdata.dev — create database → load 64 rows → build BM25 + cosine vector indexes → confirm via list_indexes → delete. BM25 built in 3.1s, vector in 0.7s, both ready, both correctly named. create_index against a nonexistent table raised RuntimeError rather than reporting phantom success. Test databases were cleaned up.

123 unit tests (tests/test_indexes.py, 26 new) cover the job-failure path, partially_succeeded, timeouts, the 201 inline path, enum-vs-wire-string status, read-probe skipping, and every validation branch.

Known issue — not introduced here

A plain vector index is created correctly (status=ready, right metric and column) but the planner does not rewrite cosine_distance(…) ORDER BY … LIMIT k to USearchExec; it full-scans. Reproduced identically with an index created by hotdata indexes create with no wrapper involved, so it is not a client bug. Held across 1536 dims/200 rows, a 2+ minute wait, and all three table-reference forms. BM25 on the same table works, so it is vector-rewrite-specific. Managed-database indexes register under the internal __db_<id>.public.docs label while the plan sees default.public.docs — a suspected, unconfirmed registry-key mismatch. Worth a separate engine issue.

Separately, the could not detect dimension failure noted during earlier spikes is runtimedb #934 (scan-based detection is unreliable for a DuckLake List column), already fixed upstream by pinning the width from the catalog.

Release

Ships as 0.10.0 (additive; hotdata_framework/ is 291 insertions and zero deletions). hotdata-langchain pins >=0.9.0 and will need a bump to consume this.

Follow-ups

• No list_indexes / delete_index wrappers, so an idempotent ensure_index still needs the raw API.
• async_after_ms is not exposed; polling makes it redundant.

[claude]

nit: .actual_instance is accessed unguarded, and every test that exercises this line mocks the job as SimpleNamespace(actual_instance=...) — so the suite gives no evidence about the real shape of JobStatusResponse.result. If that field is ever the model directly rather than a oneOf wrapper (or a dict), a successful build raises AttributeError from here instead of returning a result. Cheap to make shape-agnostic:

Suggested change

	built = job.result.actual_instance if job.result is not None else None
	built = getattr(job.result, "actual_instance", job.result)

(not blocking)

[claude]

nit: this is an allowlist of terminal states, so any terminal status not in _JOB_TERMINAL — a cancelled state that appears later, or any new state the server adds — is treated as "still running" and burns the whole timeout_s (300s by default) before surfacing as TimeoutError rather than a RuntimeError naming the actual outcome. The comment at line 79 asserts jobs have no cancelled state, which is true of today's server but is exactly the kind of thing that changes without a client release. Checking the non-terminal set instead fails fast and degrades better:

Suggested change

	if last.status in _JOB_TERMINAL:
	if enum_value(last.status) not in _JOB_RUNNING: # {"pending", "running"}

(not blocking)

[claude]

nit: two things about this block.

1. It is byte-for-byte the same construction as lines 619–630 except for status. A single local helper (closing over full_name/schema/table/resolved_name/index_type/columns/metric/embedding_provider_id and taking status) would keep the two paths from drifting — e.g. a future field added to CreateIndexResult is easy to add in one place and forget in the other.
2. status="pending" is synthesized even though the server just told you what it is on submitted.status. If the server ever answers 202 with running, the result misreports. enum_value(submitted.status) is the faithful value and _submitted() in the tests already sets it.

(not blocking)

[claude]

super nit: these restate IndexType and VectorMetric (lines 65 and 72), so adding a kind or metric means editing two places and a miss silently rejects a valid value at runtime while type-checking clean. Deriving them removes the second source of truth:

Suggested change

	_INDEX_TYPES = frozenset({"sorted", "bm25", "vector"})
	_VECTOR_METRICS = frozenset({"l2", "cosine", "dot"})
	_INDEX_TYPES = frozenset(get_args(IndexType))
	_VECTOR_METRICS = frozenset(get_args(VectorMetric))

(needs get_args added to the typing import) (not blocking)

[claude]

super nit: description is grouped with the genuinely vector-specific options here, but unlike metric/dimensions/embedding_provider_id/output_column it carries no vector semantics — it reads as generic index metadata. Rejecting it on bm25/sorted is a hard error on a call the server would accept, and if the server does persist it (or starts to) for those kinds, callers have to wait for a client release to use it. Being strict about combinations the server silently ignores is the right instinct; I'd just double-check description is actually one of them rather than stored-and-unused.

Related: description is the one parameter in the signature that the docstring never explains, so its meaning is only discoverable via the rejection message. (not blocking)

[claude]

super nit: "columns then holds the generated embedding column instead" holds only when the server returned the built index. On the wait=False path and the succeeded-but-no-result fallback, provider-backed mode returns columns == [source_text_col] and source_column == source_text_col — the same value in both fields, not the embedding column. The sentence at the end of the docstring hedges this ("echo the requested values when the server does not return the built index"), but a reader who takes the earlier paragraph literally will use columns[0] as the embedding column name and get the text column. Worth making the exception explicit in the source_column paragraph itself. (not blocking)

[claude]

Additive, well-scoped, and consistent with the existing managed-table methods: same database resolution via _as_managed_database, same ApiException -> RuntimeError wrapping, and _poll_job mirrors _poll_query_run. Polling the build job instead of trusting the 202 is the right call, and the isinstance dispatch follows the _execute_sql_once precedent. CONTRACT/CHANGELOG/README and the export list are all updated together, and the tests cover the failure, partially_succeeded, timeout, inline-201, and validation branches.

Six non-blocking comments inline. The one worth a look before merge is the unguarded job.result.actual_instance on the success path — every test that reaches it mocks that shape, so the suite can't tell you whether it matches the real JobStatusResponse.

Note that CI (CI / Test (Python 3.12), Verify changelog matches version bump) had not reported when this review ran, so I have not seen the test results.

[claude]

Docs-only changes since the last review (CHANGELOG release date, README wording) — both accurate, and the vector-planner caveat is a useful thing to have written down in the changelog rather than only in the PR description.

No new findings. The prior threads are all nits/super nits and stay non-blocking; not re-raising them.

Note: I could not confirm that JobStatus is a str-mixin enum (the hotdata dependency isn't installed in the review checkout), which is what makes the raw membership test at client.py:802 work where line 611 uses enum_value. tests/test_indexes.py covers it with the real enum, but CI / Test (Python 3.12) hadn't reported when this review ran — worth a glance at that check before merging.

[claude]

nit: switching wait=False to report enum_value(submitted.status) left the docs behind — three places still promise "pending" unconditionally, and the new test asserts "running", so a reader following the docs writes if result.status == "pending" and misses the accepted-but-already-running case:

• here ("A caller that passed wait=False always gets \"pending\"")
• client.py:526 (the result then carries status="pending" and a job_id)
• CONTRACT.md:67 and CHANGELOG.md:34

The truthful statement is now "whatever status the server reported when it accepted the job — pending in practice, but read it rather than assume".

Suggested change

	inline. A caller that passed ``wait=False`` always gets ``"pending"`` and
	owns checking the job's outcome.
	inline. A caller that passed ``wait=False`` gets whatever status the server
	reported when it accepted the job (``"pending"`` in practice) and owns
	checking the job's outcome.

(not blocking)

[claude]

Prior nits addressed: shape-agnostic job.result read (with a test for the bare-model case), get_args-derived _INDEX_TYPES/_VECTOR_METRICS, the duplicated result construction folded into requested_result() now reporting the server's own status, and the columns/source_column echoing exception spelled out in CreateIndexResult. One leftover doc nit inline; nothing blocking.

Note: CI's test job had not reported when this review ran, so I have no test results to cite.