Skip to content

Deliver the end-to-end MAF demonstration (story 4.2) - #15

Merged
fabbrik merged 1 commit into
mainfrom
feat/4-2-maf-demo
Sep 23, 2026
Merged

fabbrik merged 1 commit into
mainfrom
feat/4-2-maf-demo

Conversation

@fabbrik

@fabbrik fabbrik commented Sep 23, 2026

Copy link
Copy Markdown
Owner

A runnable sample that carries one task from a failed approach, through a verified fix, to a lesson a later run reuses.

dotnet run --project samples/AgentExperience.Sample.EndToEnd

On a fresh clone this needs nothing but the .NET 10 SDK — no Docker, no database, no credentials, no network. AGENTEXPERIENCE_SAMPLE_POSTGRES swaps in the real PostgreSQL adapters and changes nothing else; an unreachable database exits 1 rather than falling back and narrating a stage that did not happen.

Built to be honest rather than impressive

Reuse feedback is recorded with benefit Unknown, and the transcript says "nothing moved — and that is the correct answer". Exposure alone moves no confidence score, and a demo that fabricated a human assessment to make the number rise would misrepresent the one property that makes this library worth trusting. The report names story 4.4 as where measurement happens and claims no model improvement anywhere.

Stage 3 states a trust boundary the library cannot enforce. Which verification round is closed is the host's claim, not the library's finding — the aggregator cannot tell a genuine fix-and-rerun from a host parking a failure in a round it simply never closes. The sample stages the honest version and says so, rather than letting a reader infer a guarantee that is not there.

Run A drives the capture service directly, because UseExperienceCapture records one MAF invocation as exactly one attempt (CaptureScope.cs:239, :285-311), so a multi-attempt run is not reachable through the middleware. The sample says this plainly instead of presenting hand-driven capture as the normal integration path. Logged as its own story.

Verification reads the stores, not the narration

A golden transcript is checked in and compared byte-for-byte, and the tests read back the persisted record, the captured run's tool-call arguments, and the feedback ledger. That combination is what makes a stage that printed without running fail.

The secret tool argument is asserted absent from the render, the captured run, the persisted record, the injected block and the ledger row — and present-but-empty in the captured arguments, so a policy that dropped the field instead of redacting it would not pass either.

Determinism is asserted under invariant, de-DE and tr-TR; the transcript emits \n explicitly so it is byte-identical across platforms, not just across two runs on one box. CI now runs the sample twice and diffs both runs against the golden file.

Review

Two reviewers. The mutation reviewer found 11 of 16 mutations survived the original 9 tests: deleting the reuse-feedback call entirely, moving the secret field to the allowed list (printing a live token beside a sentence claiming it was redacted), and making the failing attempt succeed all passed. The suite compared literal strings to literal strings. That is what the golden transcript and the store assertions replace.

Tests: 1096 → 1128, zero warnings under TreatWarningsAsErrors.

🤖 Generated with Claude Code

Add a runnable sample that carries one task from a failed approach through a
verified fix to a lesson a later run reuses. `dotnet run` on a fresh clone
needs nothing but the .NET 10 SDK: no Docker, no database, no credentials, no
network. Setting AGENTEXPERIENCE_SAMPLE_POSTGRES swaps in the real PostgreSQL
adapters and changes nothing else; an unreachable database exits 1 rather than
falling back and narrating a stage that did not happen.

The sample is built to be honest rather than impressive. Reuse feedback is
recorded with benefit Unknown and the transcript says "nothing moved -- and
that is the correct answer", because exposure alone moves no confidence score
and claiming otherwise would misrepresent the one property that makes the
library trustworthy. The report names story 4.4 as where measurement happens
and claims no model improvement anywhere.

Stage 3 states a trust boundary the library cannot enforce: which verification
round is closed is the host's claim, and the aggregator cannot distinguish a
genuine fix-and-rerun from a host parking a failure in a round it never closes.
The sample stages the honest version and says so, rather than letting a reader
infer a guarantee that is not there.

Run A drives the capture service directly because `UseExperienceCapture`
records one MAF invocation as exactly one attempt, so multi-attempt capture is
not reachable through the middleware. The sample says this plainly instead of
presenting hand-driven capture as the normal path.

Verification is by golden transcript and by reading the stores, not by reading
the narration: the persisted record, the captured tool-call arguments and the
feedback ledger are asserted directly, so a stage that printed without running
fails. The secret tool argument is asserted absent from the render, the
captured run, the record, the injected block and the ledger, and present but
empty in the arguments. Determinism is asserted under invariant, de-DE and
tr-TR, and CI now runs the sample twice and diffs it against the golden file.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
@fabbrik
fabbrik merged commit 1d65808 into main Sep 23, 2026
1 check passed
@fabbrik
fabbrik deleted the feat/4-2-maf-demo branch September 23, 2026 00:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant