Skip to content

feat(ingest): --continuous flag on new-datasource (buckets) - #249

Merged
eddietejeda merged 2 commits into
mainfrom
feat/ingest-continuous-flag
Aug 8, 2026
Merged

feat(ingest): --continuous flag on new-datasource (buckets)#249
eddietejeda merged 2 commits into
mainfrom
feat/ingest-continuous-flag

Conversation

@eddietejeda

Copy link
Copy Markdown
Contributor

Summary

Adds a --continuous flag to hotdata ingest new-datasource (buckets/filesystem only). It rides through to the datasource request body's continuous field, marking the datasource for the native cross-workspace continuous-sync scheduler — the dlthubworker-sync CronJob that hits /internal/ingest/jobs/sync-all every ~60s and incrementally re-drains every continuous filesystem datasource (append-only on newly-arrived object keys).

Until now the flag existed on the worker/API contract but there was no CLI way to set it — you had to hand-craft a raw POST /ingest/sources. This closes that gap.

Why

We land external telemetry into S3 buckets and read it back through Hotdata (e.g. the MQTT and the new OTel otel.hotdata.devs3://hotdata-labs-otel-telemetry-…/otel pipelines). Those buckets get new objects continuously, so their datasources should auto-refresh rather than needing a manual trigger-import each time. --continuous is the one-liner that opts a bucket datasource into that.

hotdata ingest new-datasource --service buckets \
  --bucket-url s3://hotdata-labs-otel-telemetry-857481120914/otel \
  --format jsonl --glob '**/*.json' --name otel_telemetry \
  --config @creds.json --continuous

Changes

  • src/commands/ingest.rs--continuous arg on new-datasource, wired into the created request; unit test create_request_filesystem_carries_continuous_flag asserts it rides through (and is off by default).
  • src/client/ingest.rscontinuous: bool on IngestRequest (skipped when false, so the wire contract is unchanged for everyone else).
  • skills/hotdata/SKILL.md — document the flag.

Notes

  • Filesystem-only. The worker only honours continuous for family=filesystem; it's ignored for SQL/REST/Iceberg/etc. (matching the worker's _validate_ingest).
  • Backward compatible — the field is omitted from the body unless --continuous is passed.

Marks a bucket datasource for native continuous sync: the worker's scheduler
re-runs it incrementally, appending only newly-arrived objects. Adds
IngestRequest.continuous (serialized only when set) + the --continuous flag,
wired into the flag-driven filesystem builder and the wizard-skip check.
Server ignores it for non-filesystem families.

Test: --continuous rides through to the request body; default off is omitted
from the JSON.
@eddietejeda
eddietejeda requested a review from a team as a code owner August 8, 2026 17:16
@eddietejeda
eddietejeda requested review from zfarrell and removed request for a team August 8, 2026 17:16
@eddietejeda
eddietejeda merged commit 06ee43c into main Aug 8, 2026
27 checks passed
@eddietejeda
eddietejeda deleted the feat/ingest-continuous-flag branch August 8, 2026 18:05
eddietejeda added a commit that referenced this pull request Aug 8, 2026
* chore: Release hotdata-cli version 0.22.0

* chore: dedupe --continuous changelog line (#248/#249 same change)

---------

Co-authored-by: Eddie A Tejeda <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant