Skip to content

Unify panel queries behind a single Prometheus-compatible datasource - #37

Open
sudosalim wants to merge 23 commits into
mainfrom
datasource_gateway_proxy
Open

Unify panel queries behind a single Prometheus-compatible datasource#37
sudosalim wants to merge 23 commits into
mainfrom
datasource_gateway_proxy

Conversation

@sudosalim

@sudosalim sudosalim commented Jun 3, 2026

Copy link
Copy Markdown
Collaborator

The plugin previously spoke to three datasources (Prometheus, a forked couchbase-datasource, and a ProxyPrometheus reverse proxy for snapshot comparison), with panel-building code branching per datasource and a
PromQL→SQL++ translator living inside the plugin backend. This collapses all of that into one datasource served by a standalone sidecar: the frontend always speaks PromQL, and the gateway decides where the data actually comes from.
The plugin backend no longer participates in querying at all, it manages settings, snapshot metadata, and datasource reconciliation.

New service: datasource-gateway/

A standalone Go service exposing a strict superset of the Prometheus HTTP API. Per query it routes by
snapshot:

  • Prometheus-backed: PromQL passes straight through to Prometheus/Mimir via a streaming reverse proxy.
  • Couchbase-backed: PromQL is translated to SQL++ and evaluated by the real Prometheus engine over the fetched samples, so rate/irate/increase and every other construct match a passthrough by construction.
  • Overlap comparison (job=~"a|b"): fans out per snapshot, queries each over its own window, and shifts timestamps to a shared t=0 axis so runs of different lengths can be overlaid.

Because it's API-compatible, a deployment that needs neither Couchbase nor overlap can point straight at Prometheus and skip the gateway entirely.

Routing is driven by the snapshot's metadata document:

Field Meaning
ts_start Start of the snapshot window.
ts_end End of the window. The sentinel "now" (or absent) marks a still-running snapshot, whose window ends at the current time and is re-resolved rather than cached.
store "couchbase" routes through the SQL++ path; anything else, including absent, serves from Prometheus.

Metadata is resolved once per snapshot and cached (with TTL), with concurrent lookups for the same snapshot sharing a single fetch.

Plugin and frontend

  • Panels are PromQL-only; the per-panel datasource field and the DataSourceToggle UI are gone, as is utils.cbquery.
  • Overlap comparison uses the same datasource as everything else. The gateway does the time alignment server-side.
  • New datasourceCapabilities service reads /config/datasources and gates gateway-only affordances (currently the overlap toggle), degrading closed if the endpoint is unavailable.
  • New gateway block in app settings (enable / URL / overlap), configurable from the Configuration page or provisioning.
  • The reconciler is the single writer of datasources. It owns the prometheus UID and deletes any pre-existing cbdatasource, so existing deployments clean themselves up on the first reconcile pass.

Configuration and deployment

The gateway is configured by layering, in increasing precedence: built-in defaults → configs/datasource-gateway/config.yamlDSG_* environment variables → section.field=value arguments. Secrets should go through the environment; an argument would be visible via ps/docker inspect.

The gateway is the generalisation of the https://github.com/m-tarhon/SyncedApp  implementation.

Comment thread datasource-gateway/internal/querybuilder/querybuilder.go
Comment thread datasource-gateway/internal/router/router.go
@sudosalim
sudosalim force-pushed the datasource_gateway_proxy branch from 15ed5b9 to 570194c Compare July 17, 2026 18:06
@sudosalim
sudosalim force-pushed the datasource_gateway_proxy branch from 43ea877 to e9d5ad0 Compare July 29, 2026 15:02
@sudosalim
sudosalim marked this pull request as ready for review August 6, 2026 17:49
@sudosalim sudosalim changed the title Implement a standalone datasource gateway Unify panel queries behind a single Prometheus-compatible datasource Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants