[b7r6]

What and why

Setting OTEL_EXPORTER_OTLP_ENDPOINT (or NL_OTEL_ENDPOINT) to an https:// URL stopped NativeLink before it served anything: opentelemetry-otlp was built without a TLS feature and refuses to build a tonic exporter for HTTPS without one. The only collector a process could export to was one it reached in plaintext, so a worker outside the cluster could not send telemetry to the cluster's collector.

This enables tls-ring on opentelemetry-otlp in nativelink-util (ring is already the rustls provider, so no new crypto backend) and reads the OpenTelemetry specification's certificate variables, which the Rust SDK does not read itself:

• An https:// endpoint gets server-authenticated TLS against the system trust store. tonic's bare ClientTlsConfig::new() trusts nothing at all, so the roots are enabled explicitly; a plaintext endpoint gets no TLS configuration, so a host without a certificate bundle still exports.
• OTEL_EXPORTER_OTLP_CERTIFICATE names a private CA, and then only that CA is trusted.
• OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE and OTEL_EXPORTER_OTLP_CLIENT_KEY name the identity for mTLS and have to be set together.
• The per-signal forms (OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE and so on) win over the general one, as the specification says.
• A plaintext endpoint never gets a TLS configuration, whatever the certificate variables say; a deployment can send one signal over TLS and another in plaintext on a host without a certificate bundle.
• With NL_OTEL_ENDPOINT, the three exporters share one channel built with the general variables, as before. An exporter handed a channel ignores its own TLS configuration, so when any per-signal certificate variable is set each signal gets a channel of its own, built with that signal's resolution of the variables. A plaintext NL_OTEL_ENDPOINT builds no TLS configuration either way.
• An exporter handed that channel sends on it and ignores the SDK's endpoint, so the SDK endpoint's certificate variables are not examined for it either: a stale OTEL_EXPORTER_OTLP_ENDPOINT pointing at an https:// collector with a missing CA file no longer stops a process that connects in plaintext through NL_OTEL_ENDPOINT.
• A file that cannot be read, or half an identity, stops the process at startup with an InvalidArgument naming the variable. A malformed NL_OTEL_ENDPOINT now does the same instead of panicking.

Both docs pages that list the telemetry variables gained rows for the three new ones, say that an https endpoint gets TLS, and say that a per-signal certificate variable gives that signal its own NL_OTEL_ENDPOINT channel.

Fixes TraceMachina#2983

How was this verified?

• The pre-change binary in my Bazel output (bazel-bin/nativelink, built from main at 283ff77) contains the string uses HTTPS but no TLS feature is enabled once; the binary built from this branch does not contain it, so the refusal is compiled out.
• Ran the issue's repro against the new binary with a memory-store config: OTEL_EXPORTER_OTLP_ENDPOINT=https://localhost:4317 nativelink config.json5 logs Ready, listening on 127.0.0.1:50971, is still up after eight seconds with the port in ss -ltn, and shuts down cleanly on SIGTERM. Nothing was listening on 4317, which is the issue's point: the process no longer dies at startup.
• OTEL_EXPORTER_OTLP_CERTIFICATE=/nonexistent/ca.pem stops it with Could not read /nonexistent/ca.pem, which OTEL_EXPORTER_OTLP_CERTIFICATE names: No such file or directory; OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE without the key stops it with a message naming both variables.
• New unit tests in telemetry.rs (all #[serial(env)]): building a span exporter for an https:// endpoint succeeds (this one does not even compile without the feature, since with_tls_config is feature-gated, and would fail at build() on the old binary); no variables yields no TLS config; a CA file yields one; a client certificate and key yield one; the per-signal variable wins over the general one; a client certificate without its key and a missing file are both InvalidArgument naming the variable; an https:// per-signal endpoint gets a config while the default plaintext endpoint does not.
• Review follow-up, the per-signal channel: with NL_OTEL_ENDPOINT=https://localhost:4317, OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE at a dummy PEM and OTEL_EXPORTER_OTLP_CERTIFICATE at a missing file, the TRACES channel builds while the METRICS channel, and the whole per-signal set, fail with InvalidArgument naming OTEL_EXPORTER_OTLP_CERTIFICATE. The METRICS error precedes name resolution so it holds without DNS; the TRACES success needs DNS and is skipped the way channel_present_when_valid_endpoint_set skips. A second test checks that without per-signal variables (an empty one included) the signals share one channel, and that without the endpoint variable they get none. Before the fix, the first test's TRACES assertion fails: the shared channel read the general variable and errored on the missing file.
• Review follow-up, mixed deployments: with OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE and _CLIENT_KEY at dummy files and OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://collector:4317 while the general endpoint stays the plaintext default, exporter_tls_config("LOGS") is None and exporter_tls_config("METRICS") is Some. Before the fix the LOGS call returned a config, and the exporter handed it to tonic, which loads the trust roots and dies on a host with no CA bundle; the existing CA-file test asserted that wrong behaviour and now asserts None.
• Review follow-up, scheme case: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=HTTPS://collector:4317 yields a config from exporter_tls_config("TRACES") (the SDK accepts the uppercase scheme, and the old starts_with("https://") would have left the trust store empty), and a value that does not parse as a URL yields none.
• Review follow-up, supplied channel: with OTEL_EXPORTER_OTLP_ENDPOINT=https://unused.example:4317 and OTEL_EXPORTER_OTLP_CERTIFICATE at a missing file, choosing the transport for an exporter without a channel is InvalidArgument, and with a load-balanced channel it succeeds (the channel half needs DNS, and skips itself where there is none). Test supplied_channel_skips_the_sdk_endpoints_tls_configuration.
• cargo test -p nativelink-util --profile=smol --lib telemetry: 16 passed. cargo test -p nativelink-util --profile=smol --test telemetry_test: 6 passed.
• bazel test //nativelink-util:unit_test //nativelink-util:integration: 23 of 23 pass, with the rustfmt and clippy aspects.
• Docs: lint:anchors and lint:snippets pass; vale is clean on the changed pages. Biome does not process .mdx, and typos is not on this machine's path, so those two ran only as far as CI will run them.
• Not verified: an actual TLS handshake against a collector with a real certificate. The TLS path is tonic's own, the same one the gRPC stores already use, and this change only decides which ClientTlsConfig to hand it.
• Cargo.lock and MODULE.bazel.lock did not change: tonic already carries tls-ring in the graph.

Risk

Low on existing deployments. A plaintext endpoint takes exactly the path it took before: no TLS configuration is built, even with the certificate variables set, so nothing new runs on a host without a certificate store, and NL_OTEL_ENDPOINT still builds one shared channel unless a per-signal certificate variable is set. The new code runs only when an endpoint is https:// or a certificate variable is set, both of which were fatal before. The one behavioural change on an existing deployment is that a malformed NL_OTEL_ENDPOINT now exits with an error instead of a panic; the process stops at startup either way. No new crate is linked, since ring and rustls were already in the binary.

AI assistance

An agent (Claude Code) wrote the change and tests under b7r6's direction; b7r6 reviews every line before it goes upstream.

🤖 Generated with Claude Code

[b7r6]

Baked here; shipped upstream as TraceMachina#3023.