runtimedb: vector index creation fails with 'could not detect dimension' on some tables

[rohan-hotdata]

Building a vector index on an existing embedding column fails on some tables with
could not detect dimension for '<column>', and reproduces on retry for the same table
while structurally similar tables index successfully. The trigger was not identified.

This blocks the HNSW fast path for HotdataVectorStore on any table that lands in this
state, and there is no client-side workaround (see "Why this cannot be fixed in the client"
below).

What happened

While verifying the vector fast path for HotdataVectorStore (hotdata-langchain 0.4.0),
we created cosine vector indexes on seven managed tables in a live workspace
(api.hotdata.dev). Six succeeded. One failed with:

could not detect dimension for 'embedding'

The failure reproduced on retry against the same table. Recreating an equivalent table
from scratch and indexing it succeeded, so the problem is table state, not the schema.

All seven tables had the same relevant shape:

• column embedding, arrow type List(Float32), 1536 dimensions (OpenAI
  text-embedding-3-small)
• written through load_managed_table(..., mode="upsert", key=["id"]) from a pyarrow
  table serialised to parquet
• index request: index_type="vector", metric="cosine", columns=["embedding"]

How the failure surfaces

Index creation is an async job. The create_index call returns successfully with a job id
and status pending; the error only ever appears on the job record, in
JobsApi.get_job(id).error_message. A caller that does not poll sees a successful create
and an index that never materialises.

Hypotheses tested and ruled out

Each was checked by building a table exhibiting only that property and indexing it. All four
indexed successfully, so none of these is the trigger on its own:

Hypothesis	Result
Parquet fragmentation from repeated upserts	ruled out — a table with several successive upserts indexed fine
A prior mode="delete" load leaving tombstones	ruled out — a table with a delete applied indexed fine
Promoted typed metadata columns alongside the vector column	ruled out — indexed fine
The List(Float32) column type itself	ruled out — it is indexable; six tables with that exact type succeeded

We stopped after six controlled reproductions rather than continue guessing. Whatever the
trigger is, it is some accumulated table state that the dimension-detection step does not
handle, and it is not any of the obvious candidates above.

Why this cannot be fixed in the client

hotdata-framework 0.10.0's create_index accepts a dimensions argument, but it does not
apply here. Per its own contract, when indexing an existing vector column the width is
read from the data; dimensions applies only to the provider-backed path, where the engine
embeds a source text column itself. So passing dimensions=1536 — which we were doing — is
ignored on this path, and there is no argument a caller can supply to bypass detection.

That means every consumer that indexes an existing embedding column is exposed:
hotdata-langchain's HotdataVectorStore, and anything built on
hotdata-dlt-destination's write side once its embedding adapter lands.

What would resolve this

Either of:

1. Make detection robust to whatever table state defeats it. Ideally the detection step
   reports what it saw when it fails (file count, first row inspected, the type it read)
   rather than a bare "could not detect".
2. Let the caller assert the width on the existing-column path too, so dimensions= is
   honoured as an override rather than ignored. This is the smaller change and unblocks
   consumers immediately, but leaves the underlying detection bug in place.

A better error message is worth having regardless of which is chosen. The current text names
the column but nothing about why detection failed, which is what made this take six
reproductions to not diagnose.

Context

• Verified engine behaviour for the vector fast path:
  docs/engine-contract.md
  (section "The vector index is also reached without naming it")
• Consumer that hits this: HotdataVectorStore, shipped in hotdata-langchain 0.4.0 (HotdataVectorStore Phase 1 (MVP): writes, similarity search, get_by_ids, delete, from_texts #48)
• Self-provisioned index creation (Phase 3 of Epic: HotdataVectorStore — LangChain's VectorStore primitive backed by Hotdata #47) will call create_index on exactly this
  path, so this bug will surface to LangChain users rather than just to us.

[rohan-hotdata]

Reproduced again while verifying HotdataVectorStore.create_index() before the 0.5.0 release.
Three things have changed since this was filed: there is now a preserved reproduction, three
more candidate triggers are ruled out, and the failure is reachable by users of this package
rather than only by us probing.

There is a live reproduction, and it is being kept

public.documents in managed database dbidh4tn5esw2roy7zg4sh1fqv8rov (label
langchain_vectorstore_demo, api.hotdata.dev) fails reproducibly. It is deliberately left
in the failing state so the engine side has something to inspect — please do not clean up
that database.

Nothing about it looks wrong through the surfaces a client can see:

rows: 8
columns: id Utf8View | content Utf8View | metadata_json Utf8View
         embedding List(Float32) | neighbourhood Utf8View | beds Int64 | outdoor Boolean
indexes: []

Note the last line: after the failed build, list_indexes reports no index record at all.
The failure leaves nothing behind to retry or clean up.

Three more triggers ruled out, with controls in the same database

Each control was built in the same database as the failing table, at the same size and width
(8 rows x 1536 dimensions, List(Float32), written through load_managed_table(mode="upsert", key=["id"])), then indexed with index_type="vector", metric="cosine":

Control	Result
Fresh table, one write	succeeds
Three repeated upserts of the same ids	succeeds
Column set changed between writes (promoted columns added on the second)	succeeds

All three are things the failing table has done. None reproduces it. Combined with the four
ruled out when this was filed (parquet fragmentation, a prior delete load, promoted typed
metadata columns, the List(Float32) column type itself) that is seven candidates, none of
which is the trigger on its own.

So whatever distinguishes the failing table is not visible through SQL, information_schema,
or the control plane's index listing. That is the main reason the live table is worth keeping:
diffing it against a working table in the same database is something only the engine side can
do.

It now reaches users, not just us

HotdataVectorStore.create_index() shipped in
hotdata-langchain 0.5.0, so any LangChain
user provisioning an index for their own table can hit this.

Severity, stated plainly: nothing is corrupted and searches remain correct on an affected
table. They stay full scans, permanently, because the table cannot be indexed and there is no
client-side remedy — hotdata-framework 0.10.0's own contract is that a plain vector index's
width is read from stored data and dimensions applies only to the provider-backed path, so
there is no argument a caller can supply to bypass detection. It is a silent performance cliff
with no user-side recovery.

The release notes carry it as a known issue for that reason.

Unchanged from the original report

The two possible resolutions in the issue body still stand: make detection robust to whatever
table state defeats it, or honour dimensions as an override on the existing-column path. A
better error message is worth having either way — the current text names the column but nothing
about why detection failed, which is what has made this take seven controls to still not
diagnose.