Skip to content

Feedback & Wissenslücken, Antwortqualität, Aktualität, S3 Vectors, Admin-Statistik - #471

Merged
bl0rb merged 7 commits into
mainfrom
claude/friendly-bell-5l0vb9
Nov 17, 2026
Merged

bl0rb merged 7 commits into
mainfrom
claude/friendly-bell-5l0vb9

Conversation

@bl0rb

@bl0rb bl0rb commented Nov 17, 2026

Copy link
Copy Markdown
Owner

Summary

Implements the three planned feature packages plus two follow-ups:

1. Feedback und Wissenslücken

  • Chat answers can be marked Hilfreich, Falsch or Veraltet, optionally with a comment.
  • Weave-API stores the rating per user and returns message_id in /v1/chat and in the done SSE event.
  • Weave-API forwards anonymous quality signals to Weave-Ingest via an outbox. Signals cover feedback and unanswered "no sources" questions. They never include user identity. QUALITY_SIGNALS_ENABLED=false turns forwarding off.
  • Ingest clusters similar unanswered questions per knowledge space. It turns them into Prüfaufgaben for the owners and members of the space, together with wrong/outdated feedback, due reviews and possible contradictions. They appear on a new portal page /pruefaufgaben with a badge and a quality summary.

2. Antwortqualität sichtbar machen

  • Citation markers and source cards open the cited passage in place: the chunk is highlighted between its neighbours, with heading path and page. A new Retrieval endpoint POST /api/v1/passage serves it with the same visibility rules as search; the API proxies it as GET /v1/sources/passage.
  • Runtime sources now carry source_job_id, heading_path and chunk_index.
  • Retrieval eval harness: adds Precision@k, nDCG@k and --json output, and golden entries can carry an answer: block.
  • New python -m app.eval_cli answers in Runtime compares models on the same golden set. It scores retrieval, factual correctness (key facts, token F1, optional LLM judge) and grounding (faithfulness, citation coverage and precision) separately, plus abstention.

3. Aktualität und Änderungen

  • Review dates come from frontmatter review_due or a space interval. A reminder tick creates a Prüfaufgabe and sends the new webhook event document.review_due.
  • Section-wise version diff with a plain-language summary (/reviews/[id]/changes).
  • Topic subscriptions and a change feed on the portal page /aenderungen, including the search "Was hat sich an … geändert?".
  • Weave-Knowledge reports possible contradictions between very similar passages after indexing. This is a heuristic for human review only.

4. Amazon S3 Vectors (optional vector index, ADR 0010)

  • KNOWLEDGE_VECTOR_BACKEND / RETRIEVAL_VECTOR_BACKEND = pgvector (default) or s3vectors.
  • Knowledge mirrors embeddings best effort, with s3vectors init/sync/status.
  • Retrieval rechecks every S3 candidate in Postgres with the same rights, status and model filters, and falls back to pgvector on errors.
  • Wired through weave.yaml, compose, the Helm chart and weave_config.py check.

5. Admin-Statistik

  • New "Statistik" section on the admin overview: documents, versions, releases, storage per backend, chunks, embedding models, knowledge spaces, memory summaries, users/teams, review tasks.
  • Backed by GET /api/v1/admin/statistics (Ingest) and GET /api/v1/internal/stats (Knowledge).

Affected services

services/api, services/chat, services/runtime, services/retrieval, services/knowledge, services/ingest (backend + frontend), contracts/, deploy/, scripts/, docs/, README.md

New migrations:

  • api 0009_message_feedback
  • ingest 0062_quality_review_tasks, 0063_document_freshness

Knowledge has no schema change.

Test plan

  • services/api: pytest — 476 passed
  • services/runtime: pytest — 686 passed
  • services/retrieval: pytest — 260 passed
  • services/knowledge: pytest — 424 passed
  • services/ingest/backend: pytest — 1591 passed, 2 skipped
  • services/chat: vitest, npm run lint, npm run typecheck — green
  • services/ingest/frontend: vitest (407), npm run lint (0 errors), npm run typecheck — green
  • scripts/tests — 70 passed, 9 skipped (helm not available locally)
  • Cross-service contract check (API↔Ingest signals, Knowledge↔Ingest contradictions, Chat↔API↔Runtime↔Retrieval, Ingest frontend↔backend): no mismatches
  • helm template + scripts/check_chart_env.py (helm could not be run in the authoring environment)
  • S3 Vectors against real AWS; the PostgreSQL/pgvector paths (contradiction query, stats SQL) against a real database
  • Manual browser check of the new UI (feedback row, passage dialog, /pruefaufgaben, /aenderungen, diff page, admin statistics)

Checklist

  • Tests added or updated for the changed behaviour
  • contracts/ updated if an API or event changed (openapi.json regenerated, additions only; internal-chat.md, chunk-store.md, frontmatter.schema.json, events/document.review_due.md)
  • python scripts/weave_config.py check passes if configuration changed
  • Docs in docs/ updated if operators or users see the change (ADR 0010, betrieb.md §7.8, README)
  • No secrets, real customer data, or generated deploy/.env in the diff

Notes for reviewers

  • No SMTP: review reminders go out as Prüfaufgabe + webhook, e.g. n8n → mail/Teams.
  • Privacy: quality signals are anonymous. Knowledge owners see question text and comments, never who asked.
  • S3 Vectors rollout order: switch Knowledge first, run init + sync, then switch Retrieval. weave_config.py check flags the wrong order. Set S3VECTORS_REGION explicitly; botocore reads neither AWS_REGION nor instance metadata.
  • Helm: all chart workloads share one ServiceAccount, so with IRSA Retrieval also gets Knowledge's S3 Vectors write rights (noted in ADR 0010).
  • Eval CLI only: the model comparison has no admin UI yet.

Generated by Claude Code

Retrieval returns source_job_id with every hit and serves POST
/api/v1/passage (cited chunk plus neighbours, same visibility rules as
search). The eval harness adds Precision@k, nDCG@k, --json output and an
optional answer: block in golden files. Runtime carries source_job_id,
heading_path and chunk_index on sources and adds python -m app.eval_cli
answers: model comparison scoring retrieval, correctness and grounding
separately.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01W9t7FekLfn8nPUJLW1BkgU
Weave-API returns the assistant message id, stores helpful/wrong/outdated
ratings per user and forwards anonymous quality signals (feedback and
unanswered questions) to Weave-Ingest through an outbox. GET
/v1/sources/passage proxies the passage lookup within the caller's read
rights. The chat shows feedback buttons and a passage dialog opened from
citation markers and source cards.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01W9t7FekLfn8nPUJLW1BkgU
Ingest clusters unanswered questions and turns them, wrong/outdated
feedback, due reviews and possible contradictions into Pruefaufgaben for
the owners and members of a space. Documents get review dates (frontmatter
review_due or space interval) with a reminder tick and the
document.review_due webhook, section-wise version diffs, topic
subscriptions and a change feed. Knowledge reports possible contradictions
between similar passages after indexing.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01W9t7FekLfn8nPUJLW1BkgU
VECTOR_BACKEND=s3vectors (per service, pgvector stays the default) makes
Weave-Knowledge mirror chunk embeddings into an S3 Vectors index
(best effort; s3vectors init/sync/status CLI repairs drift) and
Weave-Retrieval take vector candidates from it. Every candidate is
rechecked in Postgres with the same rights, status and model filters as
the pgvector path, so a stale index cannot widen access; on S3 errors
Retrieval falls back to pgvector. Wired through weave.yaml, compose, the
Helm chart and weave_config check; ADR 0010 and betrieb.md 7.8 describe
rollout order, IAM actions and region handling.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01W9t7FekLfn8nPUJLW1BkgU
Weave-Knowledge serves GET /api/v1/internal/stats (documents by status,
chunks, embedding models, knowledge spaces, memory summaries, vector
backend, table sizes on PostgreSQL), guarded like reindex/status.
Weave-Ingest adds GET /api/v1/admin/statistics (admin only) with
documents, versions, releases, stored objects per backend, spaces, users,
teams, review tasks and subscriptions, and includes the Knowledge figures
(null with a reason when Knowledge is unreachable). The admin overview
shows them in a new Statistik section.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01W9t7FekLfn8nPUJLW1BkgU
…Vectors

- API: prune outbox rows after 30 days also without Ingest config, drain a
  backlog in one flush, retract queued signals (without text) when a
  conversation is deleted
- Ingest: resend pending document.review_due webhooks on the next tick,
  announce only delivered releases in the change feed, bound and cache gap
  clustering candidates per batch
- Retrieval: follow QueryVectors pagination
- Knowledge: run the S3 Vectors mirror and the contradiction check in their
  own post-index task

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01W9t7FekLfn8nPUJLW1BkgU
@bl0rb
bl0rb merged commit c438f60 into main Nov 17, 2026
11 checks passed
@bl0rb bl0rb mentioned this pull request Nov 17, 2026
3 of 6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants