diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 39fb422804..f5303ad03b 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -76,6 +76,7 @@ quiescence. It reaches Actors through the private runtime network. - [A2A agent tools](a2a-subagents.md) - [Human in the loop](human-in-the-loop.md) - [Prompt resolution](prompt-templates.md) +- [Telemetry](telemetry.md) The documents describe implemented behavior. Deferred work, including full cross-AgentInstance delegation and Dedicated agents, belongs in the diff --git a/docs/architecture/telemetry.md b/docs/architecture/telemetry.md new file mode 100644 index 0000000000..893b341224 --- /dev/null +++ b/docs/architecture/telemetry.md @@ -0,0 +1,222 @@ +# Telemetry + +Kagent exports OpenTelemetry traces from agent runtimes when a user enables +them. This document describes what a runtime produces, so consumers can rely on +it without reading runtime internals. + +## Enabling export + +The controller resolves telemetry from its own process environment and compiles +the result into each runtime revision. + +| Variable | Effect | +| --- | --- | +| `OTEL_TRACING_ENABLED` | Enables trace export for compiled runtimes | +| `OTEL_LOGGING_ENABLED` | Enables log export for compiled runtimes | +| `OTEL_EXPORTER_OTLP_ENDPOINT` | Destination, with the usual signal-specific overrides | +| `OTEL_EXPORTER_OTLP_PROTOCOL` | `grpc` or `http/protobuf`, with signal-specific overrides | +| `KAGENT_OTEL_CAPTURE_SENSITIVE_CONTENT` | Enables bounded prompt and response capture. Off by default | +| `KAGENT_OTEL_CAPTURE_RAW_API_BODIES` | Enables native raw provider body logging. Off by default | +| `KAGENT_OTEL_MAX_CAPTURE_BYTES` | Bytes retained per captured prompt and per captured response. Defaults to 16 KiB, ceiling 64 KiB | + +An unusable capture budget is reported as a compilation warning and replaced by +the default, so an observability setting cannot invalidate an AgentTemplate. + +The capture decision reaches every runtime as the standard GenAI instrumentation +variable, `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT`, rendered +`SPAN_ONLY` when capture is on and `false` when it is off. It is rendered +whether or not the controller exports traces, so a runtime that reaches a +collector through settings the controller did not render still follows it. The +ADK runtimes read the variable as a mode and treat a plain `true` as log records +only, so the span form is what puts `gen_ai.input.messages` and +`gen_ai.output.messages` on their model spans; the ADK Go runtime's older +`gcp.vertex.agent.llm_request` and `llm_response` payload attributes follow the +same value. The harness runtimes carry the same decision in their compiled +configuration. The variable is controller-owned: the Claude and Codex compilers +reject a `Harness.spec.env` entry with that name, and the kagent compiler +replaces one with the controller's value. One setting decides whether prompts +enter traces, and no runtime can be talked into recording them by a +user-supplied variable. + +Other `OTEL_*` variables remain available for per-Harness tuning through +`Harness.spec.env`, including `OTEL_RESOURCE_ATTRIBUTES`. + +## Conventions + +Attribute names follow the OpenTelemetry GenAI semantic conventions at version +1.41.0, and every tracer kagent creates declares that schema URL. The pin is +deliberate: 1.41.0 is the last release of the main conventions to carry the +GenAI registry, so it is the last one with a Go package of typed keys. The GenAI +conventions now live in their own repository, and the pin moves when that +repository publishes a release with a Go package. Names the conventions define +are taken from that package rather than spelled out, so the constants in +`go/pkg/tracing` cannot drift from the declared version. Only the `kagent.*` and +`a2a.*` names are kagent's own, and they are listed below. + +## The invocation span + +Each A2A `SendMessage` or `SendStreamingMessage` request opens one span in the +instrumentation scope `github.com/kagent-dev/kagent/go/adk/pkg/a2a/server`. It +represents one execution segment, and what it is called depends on whether +anything beneath it describes the agent invocation. + +For a native harness, Claude Code or Codex, nothing does: the native runtime +emits model and tool spans but no agent invocation. The request span is +therefore the GenAI conventions' `invoke_agent` operation, named +`invoke_agent `, and it is the anchor consumers should read. + +For the ADK runtime the ADK emits `invoke_agent` spans of its own, carrying +`gen_ai.agent.name` and `gen_ai.conversation.id`: one for the root agent and +one for each sub-agent it transfers to within the turn. The root one is the +invocation. The request span stays a transport span named `a2a.request` with +the same identity attributes and no `gen_ai.operation.name`, so the wrapper +never adds an invocation of its own to what the ADK reports. A consumer that +counts turns rather than agent invocations counts `kagent.invocation.segment`, +which only the request span carries, whichever runtime produced the trace. The +ADK also keeps an `invocation` span of its own beneath the request span. + +| Attribute | Meaning | +| --- | --- | +| `gen_ai.operation.name` | `invoke_agent` on a native harness request span. Absent on an ADK request span | +| `kagent.runtime` | `adk-go`, `claude`, or `codex`: the runtime that produced the spans beneath this one. The Go ADK also carries it on its resource, so its own `invoke_agent` reports it | +| `gen_ai.agent.name` | The compiled agent identity, `