feat(otel): add Datadog fan-out collector config + local traces pipeline

[NiteshDhanpal]

What this PR does

Adds the Datadog fan-out for the OTel Collector — the "collector sink" phase of the ddtrace → LGTM migration.

• New agentex/otel/otel-collector-config.datadog.yaml — opt-in collector config that duplicates traces + metrics to both the LGTM path (Prometheus/Tempo) and Datadog (datadog exporter). One OTel SDK produces the spans, so the same trace_id is exported to both backends.
• agentex/otel/otel-collector-config.yaml — adds a traces pipeline to the default local config (it previously had metrics only), so OTel traces emitted by the app are handled locally instead of dropped.

Why

Once the app stops emitting to Datadog directly via ddtrace (later in the migration), Datadog must be fed somewhere or coverage goes dark. Feeding it from the collector keeps Datadog whole while the app moves to a single OTel SDK.

Safety / rollout

• Opt-in: the fan-out lives in a separate file (mount it instead of the default). Requires DD_API_KEY (+ optional DD_SITE, default datadoghq.com). The default local collector stays keyless and unchanged.
• Uses the otel/opentelemetry-collector-contrib image already run by docker-compose (datadog exporter included).
• Migration invariant (documented in the file header): deploy + verify this fan-out in an environment before disabling app-side ddtrace there — never remove the app's Datadog source until the collector is confirmed exporting to Datadog, or that environment loses traces (this is especially important for customer VPCs).
• Reference pattern for the Helm-managed production collector (prod collector config is not in this repo).

Verification

Both YAML files parse; pipelines/exporters confirmed (metrics/traces present; datadog exporter wired into both pipelines in the fan-out file).

Follow-up (not in this PR)

Add an OTLP/Tempo trace exporter to the fan-out traces pipeline once a Tempo endpoint is available (placeholder documented in the file).

Greptile Summary

This PR advances the ddtrace → LGTM migration by adding an opt-in OTel Collector config (otel-collector-config.datadog.yaml) that fans traces and metrics to both Datadog and the LGTM stack from a single OTel SDK, and by wiring up a traces pipeline in the default local collector config that previously only handled metrics.

• New Datadog fan-out file — defines otlp receiver, batch processor, and prometheus + datadog + debug exporters; both traces and metrics pipelines are populated. Requires DD_API_KEY; DD_SITE defaults to datadoghq.com. Intended as the reference pattern for the Helm-managed production collector.
• Default local config — gains a traces pipeline (receiver → batch → debug) so OTel traces are no longer silently dropped during local development.
• Safety invariant documented — the file header calls out explicitly that this fan-out must be deployed and verified before disabling app-side ddtrace in any environment.

Confidence Score: 4/5

Safe to merge; the Datadog fan-out is completely opt-in and the default local config change is additive only.

The two files are YAML-only config changes with no code execution risk. The local config addition (traces pipeline) is trivially correct. The Datadog fan-out file is well-structured and the migration safety note is explicit. Two quality issues are worth addressing before this file is promoted to staging: the misleading TEMPO_OTLP_ENDPOINT comment that implies conditional logic not yet implemented, and the debug exporter lacking the sampling guard that the local config uses — omitting it will produce unsampled console output for every span and metric in production deployments.

agentex/otel/otel-collector-config.datadog.yaml — the TEMPO_OTLP_ENDPOINT comment and the unsampled debug exporter warrant a second look before this file is mounted in staging or used as the Helm reference.

Important Files Changed

Filename	Overview
agentex/otel/otel-collector-config.datadog.yaml	New opt-in Datadog fan-out collector config; well-structured with two issues: a misleading TEMPO_OTLP_ENDPOINT comment implying conditional behaviour that isn't implemented, and missing debug-exporter sampling that the local config applies.
agentex/otel/otel-collector-config.yaml	Adds a traces pipeline (receivers → batch processor → debug exporter) to the existing local collector config; straightforward and correct.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    App["App (OTel SDK)"]

    subgraph Default["Default Local Collector (otel-collector-config.yaml)"]
        OTLPLocal["OTLP Receiver\n(gRPC :4317 / HTTP :4318)"]
        BatchLocal["batch processor"]
        DebugLocal["debug exporter\n(verbosity: detailed + sampling)"]
        PromLocal["prometheus exporter\n(:8889)"]
        OTLPLocal --> BatchLocal
        BatchLocal -->|metrics| DebugLocal
        BatchLocal -->|metrics| PromLocal
        BatchLocal -->|traces| DebugLocal
    end

    subgraph Fanout["Datadog Fan-out Collector (otel-collector-config.datadog.yaml)"]
        OTLPFan["OTLP Receiver\n(gRPC :4317 / HTTP :4318)"]
        BatchFan["batch processor"]
        DebugFan["debug exporter\n(verbosity: normal)"]
        PromFan["prometheus exporter\n(:8889)"]
        DDExp["datadog exporter\n(DD_API_KEY + DD_SITE)"]
        OTLPFan --> BatchFan
        BatchFan -->|metrics| PromFan
        BatchFan -->|metrics| DDExp
        BatchFan -->|metrics| DebugFan
        BatchFan -->|traces| DDExp
        BatchFan -->|traces| DebugFan
    end

    App -->|"OTLP (local dev)"| OTLPLocal
    App -->|"OTLP (staging/prod opt-in)"| OTLPFan
    DDExp -->|"Datadog API"| DD[("Datadog")]
    PromFan --> Grafana[("Grafana / Prometheus")]

Loading

Prompt To Fix All With AI

Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
agentex/otel/otel-collector-config.datadog.yaml:15-16
**TEMPO_OTLP_ENDPOINT comment implies conditional behaviour that doesn't exist**

The requirements header says "Optionally set `TEMPO_OTLP_ENDPOINT` to also fan traces out to Tempo; if unset, the Tempo exporter is simply not referenced." However, no `otlp/tempo` exporter is defined in `exporters:` and no pipeline references it — the env var does nothing right now. An operator who reads this, sets `TEMPO_OTLP_ENDPOINT`, and deploys will get silent loss of the expected Tempo routing with no error from the collector. The phrase "if unset, the Tempo exporter is simply not referenced" actively suggests conditional logic that doesn't exist yet. Consider rewording to "Tempo fan-out is not yet implemented — see follow-up ticket" to avoid the false impression.

### Issue 2 of 2
agentex/otel/otel-collector-config.datadog.yaml:42-43
**Debug exporter lacks sampling — will log every span and metric in production**

The local config (`otel-collector-config.yaml`) configures `sampling_initial: 5` and `sampling_thereafter: 200` on the `debug` exporter, capping console output at about 1-in-200 records after warmup. The Datadog fan-out config, described as "the reference pattern for the Helm-managed production collector," omits those settings. With `verbosity: normal`, every span in the `traces` pipeline and every metric data point in the `metrics` pipeline will be printed to stdout. In a staging or production deployment this generates significant log volume and cost. Either add the same sampling knobs or remove `debug` from the Datadog-facing pipelines if console output isn't useful there.

_{Reviews (1): Last reviewed commit: "feat(otel): add Datadog fan-out collecto..." | Re-trigger Greptile}

Greptile also left 2 inline comments on this PR.

[greptile-apps]

TEMPO_OTLP_ENDPOINT comment implies conditional behaviour that doesn't exist

The requirements header says "Optionally set TEMPO_OTLP_ENDPOINT to also fan traces out to Tempo; if unset, the Tempo exporter is simply not referenced." However, no otlp/tempo exporter is defined in exporters: and no pipeline references it — the env var does nothing right now. An operator who reads this, sets TEMPO_OTLP_ENDPOINT, and deploys will get silent loss of the expected Tempo routing with no error from the collector. The phrase "if unset, the Tempo exporter is simply not referenced" actively suggests conditional logic that doesn't exist yet. Consider rewording to "Tempo fan-out is not yet implemented — see follow-up ticket" to avoid the false impression.

Prompt To Fix With AI

This is a comment left during a code review.
Path: agentex/otel/otel-collector-config.datadog.yaml
Line: 15-16

Comment:
**TEMPO_OTLP_ENDPOINT comment implies conditional behaviour that doesn't exist**

The requirements header says "Optionally set `TEMPO_OTLP_ENDPOINT` to also fan traces out to Tempo; if unset, the Tempo exporter is simply not referenced." However, no `otlp/tempo` exporter is defined in `exporters:` and no pipeline references it — the env var does nothing right now. An operator who reads this, sets `TEMPO_OTLP_ENDPOINT`, and deploys will get silent loss of the expected Tempo routing with no error from the collector. The phrase "if unset, the Tempo exporter is simply not referenced" actively suggests conditional logic that doesn't exist yet. Consider rewording to "Tempo fan-out is not yet implemented — see follow-up ticket" to avoid the false impression.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

Debug exporter lacks sampling — will log every span and metric in production

The local config (otel-collector-config.yaml) configures sampling_initial: 5 and sampling_thereafter: 200 on the debug exporter, capping console output at about 1-in-200 records after warmup. The Datadog fan-out config, described as "the reference pattern for the Helm-managed production collector," omits those settings. With verbosity: normal, every span in the traces pipeline and every metric data point in the metrics pipeline will be printed to stdout. In a staging or production deployment this generates significant log volume and cost. Either add the same sampling knobs or remove debug from the Datadog-facing pipelines if console output isn't useful there.

Prompt To Fix With AI

This is a comment left during a code review.
Path: agentex/otel/otel-collector-config.datadog.yaml
Line: 42-43

Comment:
**Debug exporter lacks sampling — will log every span and metric in production**

The local config (`otel-collector-config.yaml`) configures `sampling_initial: 5` and `sampling_thereafter: 200` on the `debug` exporter, capping console output at about 1-in-200 records after warmup. The Datadog fan-out config, described as "the reference pattern for the Helm-managed production collector," omits those settings. With `verbosity: normal`, every span in the `traces` pipeline and every metric data point in the `metrics` pipeline will be printed to stdout. In a staging or production deployment this generates significant log volume and cost. Either add the same sampling knobs or remove `debug` from the Datadog-facing pipelines if console output isn't useful there.

How can I resolve this? If you propose a fix, please make it concise.

[stephen-wang24]

Can you also add a note how it was tested locally?