Skip to content

stats: add UDP statsd / DogStatsD stats sinks and --stats-sink-tag - #1600

Open
bpalermo wants to merge 1 commit into
envoyproxy:mainfrom
bpalermo:up/statsd-sink
Open

stats: add UDP statsd / DogStatsD stats sinks and --stats-sink-tag#1600
bpalermo wants to merge 1 commit into
envoyproxy:mainfrom
bpalermo:up/statsd-sink

Conversation

@bpalermo

@bpalermo bpalermo commented Sep 7, 2026

Copy link
Copy Markdown

Description

This PR is related to #1607

--stats-sinks is documented in the README with a statsd example, but no NighthawkStatsSinkFactory implementation is linked into the binaries, so any --stats-sinks value aborts at startup with Didn't find a registered implementation for 'envoy.stat_sinks.statsd'. This adds a UDP statsd sink and registers it under Envoy's sink names so the documented configs work as written.

Behavior:

  • envoy.stat_sinks.statsd (envoy.config.metrics.v3.StatsdSink, UDP address form, IP literal or host name resolved at startup; tcp_cluster_name is rejected) and envoy.stat_sinks.dog_statsd (envoy.config.metrics.v3.DogStatsdSink: DogStatsD tags, optional max_bytes_per_datagram batching).
  • Counters are sent as deltas (|c) on every flush, gauges as values (|g). Nighthawk's sinkable latency statistics record nanoseconds, so every sample is sent as a millisecond timing with microsecond precision (|ms); Envoy histograms are converted per their unit.
  • Per-worker metrics are named worker.<n>.<rest> (covering both the cluster.<n>. and worker.<n>. store scopes), or carry a worker:<n> tag with DogStatsD.
  • New --stats-sink-tag key:value (repeatable; CommandLineOptions.stats_sink_tags) adds tags to every message of tag-capable sinks, e.g. to identify a run or pod.
  • Default metric prefix nighthawk (override with prefix).

Notes for Reviewers

  • Envoy's UdpStatsdSink was not reusable as-is: it keeps a thread-local writer, and Nighthawk's flush worker thread registers with TLS after sinks are created, which trips Envoy's currentThreadRegisteredWorker assert; it also forwards histogram values verbatim as |ms, wrong for Nighthawk's nanosecond statistics. The sink here shares one UDP socket across threads and batches latency samples per recording thread (sent when the batch fills, on the next flush, and at shutdown).
  • NighthawkStatsSinkFactory::createStatsSink now receives the translated typed config (it previously had no access to it), a ThreadLocal::SlotAllocator and the configured tags; include/ is documented as not a public API, and the only in-tree implementation (the test fake) is updated.
  • Testing: //test:statsd_sink_test (naming with/without tags, ns to ms conversion, counter deltas and gauges, batching and end-of-run flush against a real loopback UDP receiver, factory registration, host name resolution and config validation); test/integration/test_stats_sinks.py (full nighthawk_client runs with each sink, capturing the datagrams); //test:options_test, //test:process_test, //test:factories_test. README usage regenerated; docs/root/statistics.md and version history updated.
  • Verified end to end against an OpenTelemetry Collector statsd receiver: per-worker counts in Prometheus matched the client's JSON exactly.

--stats-sinks has been documented (with a statsd example) but no
NighthawkStatsSinkFactory implementation was ever linked into the
binaries, so any --stats-sinks value aborted at startup with "Didn't
find a registered implementation". This adds a sink and registers it
under Envoy's names so the documented configs work as written:

- envoy.stat_sinks.statsd (envoy.config.metrics.v3.StatsdSink, UDP
  address form only) and envoy.stat_sinks.dog_statsd
  (envoy.config.metrics.v3.DogStatsdSink: tags, optional
  max_bytes_per_datagram batching).
- Counters are sent as deltas (|c) on every flush, gauges as values
  (|g). Nighthawk's sinkable latency statistics record nanoseconds, so
  each sample is converted and sent as a millisecond timing with
  microsecond precision (|ms); Envoy histograms are converted per unit.
- Per-worker metrics are named worker.<n>.<rest> (both cluster.<n>. and
  worker.<n>. store scopes), or carry a worker:<n> tag with DogStatsD.
- One UDP socket is shared by all threads; with batching, latency
  samples are packed per recording thread and sent when the batch fills,
  on the next flush and at shutdown. Envoy's thread-local slots are not
  usable here because Nighthawk's flush worker thread registers after
  sinks are created.
- New --stats-sink-tag key:value (repeatable, CommandLineOptions
  stats_sink_tags) adds tags to every message of tag-capable sinks.

NighthawkStatsSinkFactory::createStatsSink now receives the translated
typed config, a ThreadLocal::SlotAllocator and the configured tags.

Signed-off-by: Bruno Palermo <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant