Skip to content

fix(chat): clean agent sources and a complete trace for streamed answers (v0.7.28) - #474

Merged
bl0rb merged 4 commits into
mainfrom
claude/wonderful-faraday-xy3vbz
Nov 18, 2026
Merged

bl0rb merged 4 commits into
mainfrom
claude/wonderful-faraday-xy3vbz

Conversation

@bl0rb

@bl0rb bl0rb commented Nov 18, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes two problems from an agent answer in Weave-Chat.

1. Raw JSON under "Quellen". The answer listed 1. Feuerwehr: [{"text": …, "source_refs": [{"document_id": …, "chunk_id": 14600}]}].

2. Misleading trace. The trace said "No retrieval call for this turn", although the agents searched and used the reranker. It also showed only router_ms.

Root causes

  • The subagent answered with a bare facts array instead of the required JSON object. _parse_subagent_answer treated that as plain text, so the raw JSON became the "fact" and its citations were lost.
  • On the stream, the only trace event went out before the agent research. Retrieval, the answering model and llm_ms/total_ms therefore never reached the UI.

Runtime

  • Parser (agents.py): accepts bare fact arrays, fenced JSON and JSON with prose before or after it. Embedded JSON must match the answer's shape; the first match found by raw_decode wins. String facts are kept. A non-list facts is ignored. An intentionally empty answer stays empty. Prose that merely contains an array stays prose.
  • A subagent that searched but answered in prose is asked once more, without tools, for the structured answer.
  • The answering model sees the cited documents next to each fact ((evidence: vpn.pdf p. 3)), not only the agent's name.
  • Agent sources are the hits a fact cites. If no fact cites a hit, all hits are kept as sources.
  • Streams end with a second, complete trace event right before sources. The opening trace leaves retrieval.hits empty; the hit list goes out once, with the closing trace.
  • Agent turns now report retrieval:
    • summed Weave-Retrieval search traces
    • used equals the reported sources
    • hits ordered by score and ranked by position
    • top_k/final_k are null
  • Agent turns also report model. Per subagent, the trace now carries name, facts, cited_hits, queries and open_points.
  • Streamed n8n turns report n8n_ms in the closing trace.

Weave-API persists the last trace (comment and test). Weave-Chat shows the new subagent details in the trace panel.

Contract: contracts/internal-chat.md has an additive entry (2026-11-18). Release commit v0.7.28.

Affected services

services/runtime, services/api, services/chat, contracts/, services/ingest/CHANGELOG.md, README.md

Test plan

  • Runtime: pytest -q services/runtime/backend/tests: 704 passed. New tests cover:
    • parser variants
    • the format re-ask
    • cited sources
    • closing trace for single-agent, multi-agent, n8n and knowledge turns
    • ranks and the answer model
  • API: tests/test_streaming_api.py: 25 passed, including a test that the closing trace is persisted
  • Chat: vitest (components/chat): 119 passed, plus typecheck and lint
  • Two independent review rounds (adversarial workflow plus code review). All confirmed findings are fixed.
  • Not checked against a real LLM. In particular, whether the evidence hints and the format re-ask improve the "Quellen" section in practice.

Checklist

  • Tests added or updated for the changed behaviour
  • contracts/ updated (additive, no version bump)
  • python scripts/weave_config.py check: n/a, configuration unchanged
  • CHANGELOG updated
  • No secrets or customer data in the diff

Notes for reviewers

  • Already stored answers keep their old trace.
  • If a stream fails with error, there is no closing trace, so the trace of that turn has no hit list. Weave-API does not persist such a turn anyway.
  • The format re-ask costs one extra LLM call, and only for a subagent that searched and then answered in prose.

Generated by Claude Code

- Runtime: a subagent answer given as a bare facts array, a fenced object
  or with a sentence before it is parsed instead of reaching the answer as
  raw JSON; facts keep their source_refs. The answering model sees the
  cited documents next to each fact.
- Runtime: streamed turns end with a second, complete `trace` event right
  before `sources` (agent research, answering model, llm_ms/total_ms).
  Agent turns report their retrieval (summed searches, hits, reranker) and
  per subagent name, queries, facts, cited hits and open points.
- Weave-API keeps the last trace (comment + test); Weave-Chat shows the new
  subagent details and the search budget.
- contracts/internal-chat.md: additive entry for both.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01W9t7FeuDDnBkiM8kKa8MN9
- Parser: JSON inside a reply must have the answer's shape (fact objects
  with text, or facts/open_points lists); the first match from any `{`/`[`
  wins (raw_decode), so trailing braces or a second object no longer leak
  raw JSON. Prose that merely contains an array stays prose, string facts
  are kept, a one-object array is unwrapped, and embedded JSON that yields
  nothing falls back to the reply as text.
- Closing trace: streamed n8n turns report n8n_ms; the multi-agent path
  reports the model its answer call returned; the agent search trace sums
  only searches that returned a trace (None otherwise); agent turns report
  top_k/final_k as None (the subagents choose them).
- Chat: the search-budget tag stays graph-only, as before.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01W9t7FeuDDnBkiM8kKa8MN9
…ng trace

- Parser: a non-list `facts`/`open_points` is ignored instead of being
  split per character; an intentionally empty fenced or prefixed answer
  stays empty; JSON starts are found with a regex instead of a
  per-character loop.
- A subagent that searched but answered in prose is asked once more (no
  tools) for the structured answer, so its facts keep their citations.
- Agent sources are the hits a fact cites (all hits if none is cited);
  the agent retrieval trace's `used` matches them, hits are ordered by
  score and ranked by position.
- The opening stream trace leaves retrieval.hits empty; the hit list goes
  out once, with the closing trace.
- Tests for the multi-agent closing trace, answer model, cited sources and
  ranks; contract and changelog updated.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01W9t7FeuDDnBkiM8kKa8MN9
@bl0rb
bl0rb merged commit 164f59d into main Nov 18, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants