Skip to content

fix(ingest): continuous datasources are not validate_only #253

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
eddietejeda wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(ingest): continuous datasources are not validate_only #253

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions src/commands/ingest.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -828,8 +828,14 @@ fn build_create_request(
}
}
};
// Adding a datasource discovers the schema only — never loads data.
req.validate_only = true;
// Adding a datasource discovers the schema only — never loads data — EXCEPT

@claude claude Bot Aug 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

super nit: skills/hotdata/SKILL.md:250 still says new-datasource "Validates credentials + discovers the schema; loads NO data", and the --continuous bullet at line 259 only describes the scheduled re-runs. After this change the --continuous add does load data up front. Worth a clause on line 259 noting the create seeds the initial load (not blocking).

// a continuous one, which is a persistent, self-loading datasource the
// scheduler keeps synced. Sending validate_only with continuous is
// contradictory (a one-off preview that is also permanently synced): the
// worker rejects the pair 422, and before it did, the datasource was
// re-run every tick but fell to the full-replace path — reloading the whole
// bucket forever. So continuous datasources are created ready to sync.
req.validate_only = !req.continuous;

@claude claude Bot Aug 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the post-create UX still assumes a validate-only add (not blocking).

With validate_only = false, the create job for a --continuous datasource now actually seeds data, but run_source (src/commands/ingest.rs:1335-1343) still polls with the verb "discovering schema" and render_datasource_added finishes with Import data with: hotdata ingest new-import --source {source} --all. For a continuous source that hint is wrong — it's self-loading and the scheduler keeps it synced. Also, the default --wait-timeout of 300s was sized for schema discovery; an initial full seed of a large bucket can blow past it and exit 2 (recoverable via the printed status --wait hint, but surprising).

Threading the continuous flag down to pick the verb ("seeding datasource") and swap the trailing hint would make the success output match what actually happened.

req.name = args.name;
req.database_id = args.database_id;
Ok(req)
Expand Down Expand Up @@ -1896,13 +1902,17 @@ mod tests {
assert_eq!(req.family, "filesystem");
assert_eq!(req.bucket_url.as_deref(), Some("s3://b/prefix"));
assert!(req.continuous); // --continuous rides through to the request body
// A continuous datasource is self-loading, so it is NOT validate_only —
// the worker 422s the pair, and it was the original full-replace bug.
assert!(!req.validate_only);

// Default is off, and it serializes only when true (skip_serializing_if).
let mut off = create_args();
off.bucket_url = Some("s3://b".into());
off.format = Some("jsonl".into());
let req_off = build_create_request(&e, off, None).unwrap();
assert!(!req_off.continuous);
assert!(req_off.validate_only); // a non-continuous add still discovers schema only
assert!(
!serde_json::to_string(&req_off)
.unwrap()
Expand Down
Loading