[BUG] Pipeline Health dashboard shows blank Cortex and Data Prepper panels with Docker Compose

[kangkyunghyun]

What is the bug?

With the default Docker Compose deployment, the Observability Pipeline Health dashboard shows no data for the Cortex and Data Prepper panels even while the stack is healthy and telemetry is being ingested.

The OTel Collector memory panel is also empty.

The underlying metrics endpoints expose data, but the Docker Compose OTel Collector configuration only scrapes its own metrics endpoint and Envoy. It does not scrape Cortex or Data Prepper before remote-writing metrics to Cortex.

The memory panel also queries otelcol_process_memory_rss_bytes, while the metric emitted by the collector is otelcol_process_memory_rss.

How can one reproduce the bug?

1. Start the default stack with fresh volumes:

   docker compose down -v
   docker compose up -d

2. Start the example canary to generate telemetry:

   docker compose up -d example-canary

3. Wait for several canary invocations.

4. Open OpenSearch Dashboards and navigate to:

   Observability Stack → Dashboards → Observability Pipeline Health
   

5. Observe that the following panels show no results:

   • Cortex Ingestion Rate
   • Cortex Active Time Series
   • Cortex Query Latency
   • Data Prepper records processed
   • Data Prepper pipeline latency
   • Data Prepper documents written
   • Data Prepper buffer metrics
   • OTel Collector Memory

6. Verify that the source metrics exist:

   curl -s http://localhost:9090/metrics |
     grep '^cortex_ingester_active_series'

   docker compose exec data-prepper \
     curl -s http://localhost:4900/metrics/prometheus |
     grep 'otel_traces_pipeline_recordsProcessed_total'

7. Query Cortex directly:

   curl -sG 'http://localhost:9090/prometheus/api/v1/query' \
     --data-urlencode 'query=cortex_ingester_active_series'

   The direct metrics endpoint contains the metric, but the Cortex query returns no series.

What is the expected behavior?

The Pipeline Health dashboard should display Cortex and Data Prepper self-monitoring metrics when running the default Docker Compose stack.

The OTel Collector memory panel should query the metric actually emitted by the collector.

What is your host/environment?

• Host OS: macOS 26.5.1, arm64
• Docker CLI: 29.4.0
• Docker Compose: v5.1.2
• Repository commit: 6e5b56e
• OpenSearch image: opensearchproject/opensearch:3.8.0
• Data Prepper image: opensearchproject/data-prepper:2.16.0
• OTel Collector image: otel/opentelemetry-collector-contrib:0.156.0

Are there any screenshots?

Screenshots can be provided showing:

• working OTel Collector throughput panels
• blank Cortex and Data Prepper panels
• blank OTel Collector memory panel

Do you have any additional context?

The Docker Compose collector currently configures the metrics pipeline as:

metrics:
  receivers: [otlp, prometheus/self, prometheus/envoy]

There are no receivers for Cortex or Data Prepper.

As a causal test, I ran a temporary OTel Collector configuration that scraped:

• prometheus:9090/metrics
• data-prepper:4900/metrics/prometheus

and remote-wrote those metrics to Cortex. After that change, the same dashboard PromQL queries returned values for Cortex and Data Prepper.

The dashboard also uses:

otelcol_process_memory_rss_bytes

but the emitted metric is:

otelcol_process_memory_rss

The Helm collector configuration already contains Cortex and Data Prepper scrape receivers, which indicates a configuration parity gap between Docker Compose and Helm.

[Circadian-agent]

I checked both claims in this issue against the tree on main. Both hold, and the second one has a wider blast radius than the report suggests.

To be clear about what this is: I confirmed the configuration, not the observed behaviour. I have not run the stack, so nothing here re-tests the reproduction steps.

1. The scrape gap is real, and compose-only

docker-compose/otel-collector/config.yaml has exactly two scrape jobs:

line 28: job_name: otel-collector        (targets: localhost:8888)
line 45: job_name: envoy-frontend-proxy

No Cortex, no Data Prepper, which matches the blank panels described.

The Helm chart already does this correctly, which makes it a usable reference rather than a second thing to design. charts/observability-stack/templates/otel-collector-configmap.yaml has eight jobs:

otel-collector, envoy-frontend-proxy, opensearch-exporter, cortex,
data-prepper, data-prepper-pipelines, kube-state-metrics, node-exporter

So the compose fix is largely a port of cortex, data-prepper and data-prepper-pipelines from the chart, adjusted for compose service names.

2. The metric name bug is NOT compose-only

This is the part worth flagging before someone fixes it. The dashboard file exists twice and the two copies are byte-identical:

docker-compose/opensearch-dashboards/dashboard-pipeline-health.yaml:39
charts/observability-stack/files/dashboard-pipeline-health.yaml:39
  query: "otelcol_process_memory_rss_bytes"

So the empty memory panel is not a Docker Compose problem at all. It affects the Helm deployment too, and a fix that only touches the compose copy leaves the chart broken.

The repo also corroborates which name is correct without needing an external source. Both copies of the alert rules already use the un-suffixed name:

docker-compose/prometheus/rules-stack/stack-alerts.yml:55
charts/observability-stack/files/rules-stack/stack-alerts.yml:55
  expr: otelcol_process_memory_rss{job="otel-collector"} / 1024 / 1024 > 400

So within this repo the alert rules and the dashboards disagree about the same metric, and the alert rules are the ones matching what the collector emits.

Summary for whoever picks this up

	compose	helm chart
missing Cortex / Data Prepper scrape jobs	yes	no, already correct
otelcol_process_memory_rss_bytes in dashboard	yes	yes

Happy to send a PR for either half, but I have four already open here and would rather not add a fifth to the queue unprompted. Say the word and I will pick it up.

[kangkyunghyun]

Thanks for verifying this. PR #411 already covers both the Docker Compose scrape gap and the incorrect metric name in the Docker Compose and Helm dashboard copies, including a regression test.