fix(observability): make the trace/log/metric cross-links actually work - #2041
Merged
Merged
Conversation
The "All in One" demo dashboard showed "No data" on five panels and offered
three "click exemplar dots to view traces" affordances that could never fire.
Investigating those turned up a wider problem: every cross-datasource link in
this platform has silently done nothing since the day it was written, on both
clusters.
Metrics. Six panel expressions queried `http.server.request.count`, removed by
the OpenTelemetry semconv v2 migration. The alerting rules were updated at the
time; the dashboard was not. They now use the histogram count series,
`http.server.request.duration_count`, the same shape the VMRules already use.
The P95/P50 panels were always correct because they read `duration_bucket` --
which is why exactly one panel kept rendering.
Datasource links. Three defects, each invisible:
- `datasourceName` is not a key in any Grafana schema; only `datasourceUid`
is. Trace-to-logs, trace-to-metrics, log-to-trace and the service map were
all inert.
- `tracesToLogs` (v1) is superseded by `tracesToLogsV2`, which is what
current Grafana reads, so correcting keys inside the v1 block would have
changed nothing.
- Neither datasource pinned a uid, so the operator assigned a random one and
references written as the literal strings `VictoriaLogs`/`VictoriaTraces`
matched nothing.
The uid is pinned at `spec.datasource.uid`, the field the CRD marks deprecated,
deliberately: `spec.uid` is immutable and the API server rejects it outright on
an already-created object, so Flux would fail every reconcile. Confirmed by
server-side dry-run. Renaming the objects to re-create them was the
alternative and churns live datasources for no gain.
`tracesToMetrics` also named metrics that do not exist and span attributes that
do not exist -- `http.method`/`http.status_code` rather than the semconv v2
`http.request.method`/`http.response.status_code` -- and passed `tags` as bare
strings where the schema requires key/value objects.
Exemplars. The dashboard's trace correlation was built on them and they cannot
work here: VictoriaMetrics answers `/api/v1/query_exemplars` from a hardcoded
empty stub, real support was merged upstream in 2024 and then deliberately
reverted, and the OTLP ingestion parser never reads the exemplar field at all.
No flag or histogram change helps. The dead affordances are removed and
replaced with a traces row -- a search panel and a service map on the existing
VictoriaTraces datasource -- plus a data link from the latency panel that opens
that panel filtered to slow traces over the clicked time range.
Load generator. It exported no telemetry at all, failing resource detection
with "user: Current requires cgo or $USER set in environment". The App
composition injects USER for the app containers; the plain CronJob manifest
never got one, so every trace started at the server span rather than at the
generator.
Documentation. `dashboards-and-alerts.md` reproduced the broken configuration
verbatim as the pattern to copy, so fixing only the manifests would have
guaranteed someone copies it back. `logs.md` cited the old regex and a
nonexistent metric as evidence for two rules that are themselves correct.
ADR-0010 is amended rather than edited: the driver it lists stays as the record
of what was believed, with a dated note under Consequences recording that the
linking was inert, because no gate in this repo can catch this class of defect
-- the schema validator sees a well-formed map, polaris never reads datasource
config, and the link simply never appears.
Verified: manifest gate all green, doc claims 28/49 pages, links resolve, both
datasource CRs accepted by a server-side dry-run against the live CRD.
Contributor
🔍 Rendered manifest diff — this PR vs
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What you see today
Five panels on the "All in One" demo dashboard say No data, and three more
advertise "click exemplar dots to view traces" — an interaction that cannot
happen. Investigating that turned up something wider: every cross-datasource
link in this platform has silently done nothing since it was written, on both
clusters.
Four independent defects, none visible to CI
http.server.request.count, removed by the semconv v2 migrationduration_bucketdatasourceNameis in no Grafana schema — onlydatasourceUidistracesToLogsv1 is superseded bytracesToLogsV2, which is what Grafana readsuid, so references to the literal names matched nothingflux schema validatesees a well-formed map, polaris never reads datasourceconfig, and a documentation claim only fails when it pins the exact value.
Exemplars: ruled out, permanently
The dashboard's trace correlation was built on exemplars. They cannot work here:
/api/v1/query_exemplarsis a hardcoded stub inapp/vmselect/main.goreturning
{"status":"success","data":[]}— byte-identical to what the livecluster returns
2024-07-03, with a maintainer rationale rejecting exemplars as immature
switching from exponential to classic histograms would change nothing
No flag, no version bump, no ingestion change fixes this. The dead affordances
are removed rather than left as a promise the data cannot keep.
What replaces them
A traces row — a search panel and a service map on the existing VictoriaTraces
datasource — plus a data link on the latency panel that opens that panel
filtered to slow traces over the clicked time range. Every referent verified:
the target panel id exists, the
min_trace_durationvariable is defined, andthe traces target actually reads it.
One deliberate use of a deprecated field
The uid is pinned at
spec.datasource.uid, which the CRD marks deprecated.spec.uidis immutable and the API server rejects it outright on analready-created object:
Flux would have hit that on every reconcile — a change that fixes dead links by
wedging the Kustomization that delivers them. Confirmed by server-side dry-run.
Renaming both objects so they are re-created was the alternative; it churns live
datasources for no gain. The reasoning is recorded in both files.
Also fixed
The load generator exported no telemetry at all — resource detection failed
with
user: Current requires cgo or $USER set in environment, so every tracestarted at the server span rather than at the generator. The App composition
injects
USERfor the app containers; the plain CronJob manifest never got one.Documentation
dashboards-and-alerts.mdreproduced the broken configuration verbatim as thepattern to copy, so fixing only the manifests would have guaranteed it comes
back.
logs.mdcited the old regex and a nonexistent metric as evidence for tworules that are themselves correct. ADR-0010 is amended, not edited — its
driver stays as the record of what was believed, with a dated note under
Consequences recording that the linking was inert.
Verification
./scripts/validate-manifests.sh-> All gates passed./scripts/validate-doc-claims.sh-> 28 claims / 49 page checks./scripts/validate-links.sh-> all resolveconfiguredon a server-side dry-run against the live CRD${name}across all three YAML filesNot verified, deliberately flagged
queryType: searchagainst the Jaeger-type datasource; the shape is right butno live Grafana has displayed them, and nothing in CI can.
tracesToLogsV2semantics. VictoriaLogs speaks LogsQL, not LogQL, sofilterByTraceIDmay not translate; if that direction stays empty the fix iscustomQuerywith a LogsQL expression. Noted in the file. The reversedirection — logs to trace — is the one the demo relies on and is fixed here.