feat(ingest): --continuous flag on new-datasource (buckets) - #248
Conversation
Marks a bucket datasource for native continuous sync: the worker's scheduler re-runs it incrementally, appending only newly-arrived objects. Adds IngestRequest.continuous (serialized only when set) + the --continuous flag, wired into the flag-driven filesystem builder and the wizard-skip check. Server ignores it for non-filesystem families. Test: --continuous rides through to the request body; default off is omitted from the JSON.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| /// Keep this datasource continuously synced — refreshed incrementally on a | ||
| /// schedule, appending only newly-arrived objects (buckets only) | ||
| #[arg(long)] | ||
| continuous: bool, |
There was a problem hiding this comment.
nit: the guided wizard has no matching prompt, so continuous is unreachable interactively (not blocking).
build_filesystem_interactive (src/commands/ingest.rs:527) prompts bucket URL / format / glob and leaves continuous at its Default false. On a terminal with no --service, that's the default path, so a user who wants continuous sync has to know to drop out of the wizard and pass the flag — and any_given() now makes --continuous alone hard-fail with "--service is required" rather than hinting the wizard can't do it. A one-line Confirm/select_optional("Keep continuously synced?") in the filesystem builder would close the gap; up to you whether that belongs in this PR.
| bucket_url: Some(args.bucket_url.ok_or("buckets connectors need --bucket-url")?), | ||
| file_glob: args.glob, | ||
| file_format: args.format, | ||
| continuous: args.continuous, |
There was a problem hiding this comment.
super nit: --continuous is silently dropped for every other family (not blocking).
hotdata ingest new-datasource --service postgres --continuous exits 0 and reports success with no continuous sync configured, since only the filesystem arm reads the field. That matches how --glob/--bucket-url already behave for SQL, so it's consistent — but "silently on for nothing" is a worse failure mode than an unused path hint, because the user believes sync is enabled. An early if args.continuous && entry.family != "filesystem" { return Err(...) } would make it explicit and is cheap to unit-test alongside the new test.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Reviewed cycle 2. The only new commit is the cargo fmt fix; the --continuous wiring is unchanged and correct: field is Serialize-only with skip_serializing_if, matching validate_only; any_given() updated so the flag can't be silently swallowed by the wizard; test covers both on and off (including the omitted-from-JSON case). No blocking issues.
The two prior nits (wizard has no continuous prompt; the flag is a no-op for non-filesystem families) remain open but are non-blocking — your call whether to handle here or in a follow-up.
* chore: Release hotdata-cli version 0.22.0 * chore: dedupe --continuous changelog line (#248/#249 same change) --------- Co-authored-by: Eddie A Tejeda <[email protected]>
1 participant
Lets a user mark a bucket datasource for native continuous sync from the CLI:
The worker's scheduler then re-runs it incrementally on a schedule, appending only newly-arrived objects (no full re-read).
IngestRequest.continuous— serialized only when true (skip_serializing_if), matchingvalidate_only.--continuousflag onCreateArgs, wired into the flag-driven filesystem builder + the wizard-skip check. Filesystem/buckets only; the server ignores it for other families.new-datasourcesection.Pairs with dlthubworker #167 (
continuouson the source spec + the incremental sync engine) and monopoly #1650 (the sync CronJob). Full suite green (305 unit tests).