feat(ingest): --continuous flag on new-datasource (buckets) - #249
Merged
Conversation
Marks a bucket datasource for native continuous sync: the worker's scheduler re-runs it incrementally, appending only newly-arrived objects. Adds IngestRequest.continuous (serialized only when set) + the --continuous flag, wired into the flag-driven filesystem builder and the wizard-skip check. Server ignores it for non-filesystem families. Test: --continuous rides through to the request body; default off is omitted from the JSON.
eddietejeda
added a commit
that referenced
this pull request
Aug 8, 2026
eddietejeda
added a commit
that referenced
this pull request
Aug 8, 2026
* chore: Release hotdata-cli version 0.22.0 * chore: dedupe --continuous changelog line (#248/#249 same change) --------- Co-authored-by: Eddie A Tejeda <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
--continuousflag tohotdata ingest new-datasource(buckets/filesystem only). It rides through to the datasource request body'scontinuousfield, marking the datasource for the native cross-workspace continuous-sync scheduler — thedlthubworker-syncCronJob that hits/internal/ingest/jobs/sync-allevery ~60s and incrementally re-drains everycontinuousfilesystem datasource (append-only on newly-arrived object keys).Until now the flag existed on the worker/API contract but there was no CLI way to set it — you had to hand-craft a raw
POST /ingest/sources. This closes that gap.Why
We land external telemetry into S3 buckets and read it back through Hotdata (e.g. the MQTT and the new OTel
otel.hotdata.dev→s3://hotdata-labs-otel-telemetry-…/otelpipelines). Those buckets get new objects continuously, so their datasources should auto-refresh rather than needing a manualtrigger-importeach time.--continuousis the one-liner that opts a bucket datasource into that.Changes
src/commands/ingest.rs—--continuousarg onnew-datasource, wired into the created request; unit testcreate_request_filesystem_carries_continuous_flagasserts it rides through (and is off by default).src/client/ingest.rs—continuous: boolonIngestRequest(skipped when false, so the wire contract is unchanged for everyone else).skills/hotdata/SKILL.md— document the flag.Notes
continuousforfamily=filesystem; it's ignored for SQL/REST/Iceberg/etc. (matching the worker's_validate_ingest).--continuousis passed.