[APMSVLS-501] feat(bottlecap): add bottlecap-test-mode binary

[lucaspimentel]

Stacked on #1201. Review #1201 first.

Overview

Adds a second [[bin]] target, bottlecap-test-mode, that runs the APM trace-processing surface as a long-lived HTTP server with no AWS Lambda Extension lifecycle. Reuses TraceAgent, FlushingService, and the trace/stats/proxy flushers built by bottlecap::startup::build_trace_agent, which this PR extracts from the Lambda binary (see below); the only Lambda-binary code it duplicates is init_ustr and enable_logging_subsystem (~20 lines, called out in the design doc).

Endpoints on 127.0.0.1:8126:

• /v0.4/traces, /v0.5/traces, /v0.6/stats, /info — unchanged from the Lambda binary's TraceAgent router.
• POST /flush — new, registered by a FlushRouterExtension impl attached via TraceAgent::with_router_extension(...) (the seam from [APMSVLS-501] refactor(bottlecap): preparatory work for "test-mode" binary #1201). Calls FlushingService::flush_blocking_final() in a spawned task bounded at 30s: 204 No Content on success, 500 if the flush panics, 504 after aborting a timed-out flush. flush_blocking_final expect()s on the metrics aggregator handle, so without the spawn a dead aggregator would panic the connection task and the harness would see a dropped connection rather than a status it can act on; the per-flusher HTTP timeouts also bound only individual calls, so stacked retries could otherwise outlive any sensible harness wait.

Configuration: same DD_* env vars as the Lambda binary. Notable inputs:

• DD_APM_DD_URL — redirects trace intake (the parity harness points this at the fake-intake from [APMSVLS-497][APMSVLS-498] test: add fake-intake for APM payload-level tests #1194).
• DD_SERVERLESS_FLUSH_STRATEGY — opt-in periodic flush ticker, decoupled from managed-instance mode.
• DD_TESTMODE_FUNCTION_ARN — overrides the stub ARN used for tag generation (defaults to arn:aws:lambda:us-east-1:000000000000:function:testmode).

API key is hardcoded to "stub-key" (no secrets resolver path); the parity harness fake-intake ignores auth.

bottlecap::startup extraction

This PR also promotes start_trace_agent out of src/bin/bottlecap/main.rs into a new top-level library module, bottlecap::startup, and splits it into:

• build_trace_agent — returns an unspawned TraceAgent plus a TraceAgentPipeline struct with named pub fields (flushers, trace-channel sender, shutdown token, aggregator/concentrator handles).
• start_trace_agent — thin wrapper that spawns the agent. The Lambda binary's call site is unchanged.

bottlecap-test-mode needs the unspawned agent so it can attach its /flush RouterExtension before spawning, which is why the split exists. Placed at the crate root rather than under traces/ because it wires trace, stats, proxy, lifecycle, tags, appsec, and flushing together: cross-cutting orchestration, not a trace-domain API.

This extraction was originally part of #1201 and moved here after review feedback: with no second binary in that PR, build_trace_agent had no caller and the public module looked unmotivated. Landing it next to its first consumer makes the shared-library placement self-evident. Behavior is unchanged for the Lambda binary; trace_agent.rs changes are two retargeted rustdoc links.

Feature gating

Gated behind the test-mode cargo feature via required-features = ["test-mode"], the same feature that gates InvocationProcessorHandle::noop() (added in #1201). The binary is therefore not built in default or fips builds, including by cargo build --workspace. Build/run with:

cargo build --bin bottlecap-test-mode --features test-mode
cargo run   --bin bottlecap-test-mode --features test-mode

No CI job currently builds or lints with --features test-mode, so this binary is invisible to CI as things stand. Tracked in Follow-ups below.

Shutdown ordering

signal::ctrl_c() cancels the shutdown token first (drives axum's graceful_shutdown so any in-flight /v0.4/traces request drains through the trace aggregator), then runs flush_blocking_final(). The periodic flush task selects on the same token so it doesn't leak when the listener stops.

Why a second binary

The overlap between Lambda-mode and test-mode is small (~20 lines: init_ustr, logging, config load), and the rest is intentionally different (no telemetry listener, no LWA, no logs agent, no proxy, no DogStatsD UDP, no event-bus-driven lifecycle). A second [[bin]] makes the test-mode surface explicit and structural, enforced by the compiler instead of by a runtime branch. Rejected alternatives (env-var branch, auto-detect, CLI flag, single binary with mode gating) are in the design doc.

Design doc: lucas-pimentel/docs/bottlecap-test-mode.md (local; happy to land it in-repo if reviewers prefer).

Testing

• cargo check --bin bottlecap-test-mode --features test-mode
• cargo clippy --workspace --all-targets --features default -- -D warnings (existing surface unchanged)
• cargo clippy --workspace --all-targets --features default,test-mode -- -D warnings
• cargo fmt --all -- --check
• cargo test -p bottlecap --lib — 548 passed (unchanged from [APMSVLS-501] refactor(bottlecap): preparatory work for "test-mode" binary #1201; the extraction is a pure move)
• Manual smoke test against a local fake intake on :8200:
  • POST /v0.4/traces (real msgpack span) → 200, buffered, periodic flush fired at 2s, intake received POST /api/v0.2/traces (485 bytes, DD-API-KEY: stub-key).
  • POST /flush → 204. (Predates the timeout/panic guard in 4efbc0dd; the 500 and 504 paths were verified separately against an extracted copy of the handler, not against the running binary.)
  • POST /v0.5/traces (malformed) → 500 (correct rejection from the existing v0.5 deserializer).
  • GET /info → 200 with the standard endpoints list.
  • SIGINT → TRACE_AGENT | Shutdown signal received, shutting down → Aggregator service stopped → clean exit.
  • Span dedup verified: same trace_id+span_id sent twice, second dropped by the existing dedup service.

No new unit tests in this PR. The seam (RouterExtension) and the no-op handle (InvocationProcessorHandle::noop()) are both covered by tests added in #1201; end-to-end coverage for test-mode lands as part of the parity harness (#1194 and the future apm-agent-parity-rs repo).

Follow-ups

Known gaps, called out so they are not mistaken for oversights. Neither blocks the binary from working; happy to fold either into this PR if reviewers prefer.

• CI does not exercise --features test-mode. Existing jobs run --features default and --no-default-features --features fips, and cargo build --all skips required-features targets, so nothing in CI compiles this binary. A clippy/build step passing --features test-mode would keep it green; without it, a change to library code can break this target without any job failing.
• The POST /flush guard in 4efbc0dd has no committed test. The 204/500/504 paths were verified manually, but the test-mode binary has no #[cfg(test)] module. Testing it in place needs the handler factored out of the closure, or FlushingService made injectable, so it is deliberately deferred rather than bolted on.

🤖 Generated with Claude Code

[datadog-prod-us1-5]

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 4efbc0d | Docs | View more details | Give us feedback!}