feat(tracing): add harness invocation identity, lifecycle, and bounded capture

[dhaifley]

Description

The Claude Code and Codex harnesses export traces, but a consumer cannot reliably tell which agent, conversation, task or user an exported span belongs to, and the span that anchors a turn does not always describe what actually happened. The a2a.request wrapper carries only a2a.method and, on some paths, a2a.task.state. It has no runtime marker, so a collector cannot distinguish a native harness turn from an ADK one without parsing a service name. It has no conversation or task identity, so opening the trace for a chat message means scanning a time window rather than looking one up. It ends unary requests even when execution is still running detached from the caller, and it never sets an error status, so a failed turn can read as a successful one.

This change makes invocation identity a producer contract, expressed in the OpenTelemetry GenAI semantic conventions, and makes the wrapper's lifecycle describe the real outcome. It also adds the bounded, opt-in prompt and response capture that the existing KAGENT_OTEL_CAPTURE_SENSITIVE_CONTENT switch resolves in the controller but never reached the Go wrapper. The contract is written down in docs/architecture/telemetry.md.

• A harness wrapper is an invoke_agent span; an ADK wrapper stays a transport span. For Claude Code and Codex nothing beneath the wrapper describes the agent invocation, so each A2A request opens invoke_agent <gen_ai.agent.name> with gen_ai.operation.name=invoke_agent. The ADK emits invoke_agent spans of its own, for the root agent and any sub-agent it transfers to, so its request span keeps the name a2a.request with the same identity attributes and no operation, and the wrapper never adds an invocation to what the ADK reports. A consumer counting turns counts kagent.invocation.segment, which only the request span carries. Every tracer kagent creates now declares the semantic conventions schema URL. The conventions are pinned at 1.41.0 on purpose: it is the last release of the main conventions to carry the GenAI registry, so it is the last with a Go package of typed keys, and the constants in go/pkg/tracing are taken from that package rather than spelled out so they cannot drift from the declared version. Only the kagent.* and a2a.* names are kagent's own.
• One identity for every runtime. RuntimeTelemetry carries kagent.runtime (adk-go, claude or codex), the compiled agent name, its namespace-qualified gen_ai.agent.id, and for the harnesses the gen_ai.provider.name and gen_ai.request.model the agent is compiled against. The harness compilers put it in the runtime configuration JSON they already generate, and the ADK runtime builds it from its own environment, so the same identity reaches every request span and every runtime resource. Both harness config versions move up one, since the runtime rejects a configuration it does not understand. A configuration without the section stays valid and leaves capture off, which keeps standalone harness validation working.
• The model rides on the invocation and the resource. Codex records token usage on a span that names no model, so without the compiled model a consumer cannot attribute that usage without walking the trace. The conventions allow gen_ai.request.model on an agent span when the agent is bound to one model, which a compiled kagent agent is. tracing.NewResource applies the same owned identity after OTEL_RESOURCE_ATTRIBUTES, and each adapter merges it, with service.namespace, into that variable for its native child process, preserving unrelated user-supplied attributes and replacing only owned keys. A user-supplied runtime marker is no longer required, and OTEL_RESOURCE_ATTRIBUTES remains available for other tuning.
• A request-scoped invocation handle with one completion. tracing.Invocation is started by the transport interceptor with the static identity the runtime knows before execution begins, so a request rejected during validation still reports which agent rejected it. Execution adopts it when it starts, and from that point the transport stops completing it. That is the fix for nonblocking unary requests: a2a-go v2.5.0 detaches execution from the caller's cancellation, so a response can be delivered while the task is still working, and the previous interceptor closed the span at that point. Completion is idempotent, so the executor and the response interceptor observing the same terminal event cannot double count. a2a-go runs no final interceptor callback for a streaming consumer that stopped reading, so an invocation the transport still owns is completed as abandoned when the request context ends without a quiescent event, rather than left open and never exported.
• The outcome recorded is the one execution reported. Both executors stamp gen_ai.conversation.id, a2a.task.id and kagent.invocation.segment through one shared helper, and the harness executor completes the invocation at each execution boundary with the actual task state. Failures set an OpenTelemetry error status with a safe category in error.type, never a provider response or captured content. A consumer that stops accepting events is recorded as kagent.invocation.disposition=abandoned rather than as a cancellation, and cancellation is recorded only when a client requested it, with the segment exported before the canceled event is published. Failures that never publish a task event export before the error leaves the process, and a runner panic is recorded as runtime_panic, without the panic value, before it propagates. The interceptor sets enduser.id from the gateway-established identity that UserIDCallInterceptor puts on the request, and leaves it absent when there is no trusted identity.
• Quiescent paths still export before the event leaves the process. For a terminal, input-required or auth-required outcome the executor completes and flushes the wrapper before yielding, which is earlier than the previous response-interceptor flush and matters because the gateway may suspend the Actor as soon as it sees the event. The flush keeps the existing bounded ForceFlush, three seconds by default, so an unreachable collector costs at most that budget once per segment and never converts a successful result into a failure.
• Bounded, opt-in content capture in the conventions' shape. When a user enables it, a segment records the current turn's prompt as gen_ai.input.messages and the text it produced as gen_ai.output.messages, each a JSON array holding one message with one text part, the output message carrying the finish_reason the conventions require (stop, tool_call for a segment parked for input, error, or the disposition name), bounded by KAGENT_OTEL_MAX_CAPTURE_BYTES (16 KiB default, 64 KiB ceiling), preserving UTF-8 and reporting truncation in kagent.capture.input_truncated and kagent.capture.output_truncated. The collector is allocated only when capture is on and the span is recording, and it discards further text once full, so memory stays proportional to the limit whatever the response length or delta count. Tool arguments, tool results, approval structures, the rest of the conversation and native stderr are never written to these attributes; a resumed segment records no input messages because its input is a structured decision.
• One capture switch reaches every runtime. The controller renders the standard OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT into every compiled runtime from the same setting that governs the harness wrapper, SPAN_ONLY when capture is on and false when it is off, and renders it whether or not it exports traces, so a runtime reaching a collector through its own settings still follows the decision. The mode form matters: adk-go v2.3.0 reads a plain true as log records only and puts gen_ai.input.messages and gen_ai.output.messages on its model spans only for the span modes. The ADK Go runtime already read that variable before recording its older payload attributes on model spans, but nothing set it, so those payloads were recorded whatever the controller's setting said; they now follow it. The variable is controller-owned: the Claude and Codex compilers reject a Harness.spec.env entry with that name, and the kagent compiler replaces one with the controller's value. This changes the ADK default under kagent from capturing to not capturing.
• Honest attribution across approvals. Both drivers keep the same native process while a turn waits for input, and neither native protocol offers a supported way to replace the trace context the process already received, so native work after an approval stays under the originating trace. Rather than paper over that, each execution segment gets its own invocation span carrying the same conversation and task identity, and a resumed segment records an OpenTelemetry link back to the segment that started the native turn, however many pauses later, with kagent.invocation.relationship=resume_origin. A link states a relationship; it does not reparent spans and it does not claim ownership of the token usage recorded under the originating segment.

The ADK runtime's descendant spans keep the kagent.user_id, gen_ai.task.id, gen_ai.conversation.id and kagent.app_name keys they always carried, because the Python runtimes stamp the same keys and the two should change in one step. That step, an invocation span emitted by the Python executors, is a separate change. The unused invoke_agent middleware under go/core/internal/a2a is removed, since the real one now exists.

The compiled runtime configuration is versioned and both harness versions move up one, so a controller must not run ahead of its pinned harness images. That rollout requirement, and the limitations this change does not close, are stated in docs/architecture/telemetry.md: a caller that disconnects between a2a-go dispatching execution and reading the subscription leaves that request's invocation completed from the transport side, and a request rejected by a transport interceptor before execution begins has no invocation span at all.

Not in this PR, and kept separate on purpose. The missing late-ending Codex session_task.turn and op.dispatch.turn_input spans are a native export-boundary question that needs its own reproduction against the pinned binary before any cleanup refactor, and adding gen_ai.request.model to Codex's usage span is a change in openai/codex followed by a version bump here. Neither blocks the identity and lifecycle work, and both are easier to review on their own.

Related Issues

None filed.

Testing

cd go
go build ./... && go vet ./...
go test ./...
go test -race ./pkg/tracing/... ./harness/... ./adk/pkg/a2a/... ./adk/pkg/app/... ./adk/pkg/telemetry/... ./core/internal/translator/...
make -C go lint

Change Type

Enhancement.

[krisztianfekete]

As discussed offline, we are targeting full OTel GenAI semconv compat as the goal for v1.x telemetry design. We merge this as an intermediate step, and work toward the redefined tracing contract right away.

Notes on making that easier:

• The invocation span has gen_ai.agent.name, gen_ai.conversation.id and gen_ai.task.id but is still named a2a.request and sets no gen_ai.operation.name, so every consumer has to recognise it by scope plus the harness information.
• When the runtime telemetry contract declares a native harness kind, could this span be named invoke_agent {gen_ai.agent.name} with gen_ai.operation.name=invoke_agent, leaving ADK on a2a.request so it keeps emitting exactly one invoke_agent of its own? That is the semconv shape for an agent invocation we are building towards in v1.x.

[dhaifley]

@krisztianfekete Thanks for the review. When the runtime telemetry contract declares a native harness, the request span is now invoke_agent <gen_ai.agent.name> with gen_ai.operation.name=invoke_agent, and the ADK request span stays a2a.request with no operation, so an ADK turn still emits exactly one invoke_agent, the one the ADK produces itself. Both request spans carry the same identity, and the Go ADK runtime now supplies it too, so kagent.runtime, gen_ai.agent.name and gen_ai.agent.id reach every request span and every runtime resource.

For full GenAI semconv compatibility, I moved the rest of the contract there. kagent.harness.kind became kagent.runtime, the request span's gen_ai.task.id and kagent.user_id became a2a.task.id and enduser.id, captured content is gen_ai.input.messages and gen_ai.output.messages in the conventions' message with kagent.capture.* truncation flags, and the harness request spans carry the compiled gen_ai.provider.name and gen_ai.request.model, also on the resource, which is what lets Codex usage be attributed to a model without walking the trace. Every tracer kagent creates declares the 1.41.0 schema URL, which is the last main-repository release with the GenAI registry and a Go package of typed keys. The controller also renders OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT into every runtime from the existing capture setting. A consequence is that the ADK's model payload capture now follows that setting and is off by default under kagent, where before nothing set the variable and the payloads were always recorded.