Deliver the end-to-end MAF demonstration (story 4.2) - #15
Merged
Merged
Conversation
Add a runnable sample that carries one task from a failed approach through a verified fix to a lesson a later run reuses. `dotnet run` on a fresh clone needs nothing but the .NET 10 SDK: no Docker, no database, no credentials, no network. Setting AGENTEXPERIENCE_SAMPLE_POSTGRES swaps in the real PostgreSQL adapters and changes nothing else; an unreachable database exits 1 rather than falling back and narrating a stage that did not happen. The sample is built to be honest rather than impressive. Reuse feedback is recorded with benefit Unknown and the transcript says "nothing moved -- and that is the correct answer", because exposure alone moves no confidence score and claiming otherwise would misrepresent the one property that makes the library trustworthy. The report names story 4.4 as where measurement happens and claims no model improvement anywhere. Stage 3 states a trust boundary the library cannot enforce: which verification round is closed is the host's claim, and the aggregator cannot distinguish a genuine fix-and-rerun from a host parking a failure in a round it never closes. The sample stages the honest version and says so, rather than letting a reader infer a guarantee that is not there. Run A drives the capture service directly because `UseExperienceCapture` records one MAF invocation as exactly one attempt, so multi-attempt capture is not reachable through the middleware. The sample says this plainly instead of presenting hand-driven capture as the normal path. Verification is by golden transcript and by reading the stores, not by reading the narration: the persisted record, the captured tool-call arguments and the feedback ledger are asserted directly, so a stage that printed without running fails. The secret tool argument is asserted absent from the render, the captured run, the record, the injected block and the ledger, and present but empty in the arguments. Determinism is asserted under invariant, de-DE and tr-TR, and CI now runs the sample twice and diffs it against the golden file. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A runnable sample that carries one task from a failed approach, through a verified fix, to a lesson a later run reuses.
On a fresh clone this needs nothing but the .NET 10 SDK — no Docker, no database, no credentials, no network.
AGENTEXPERIENCE_SAMPLE_POSTGRESswaps in the real PostgreSQL adapters and changes nothing else; an unreachable database exits 1 rather than falling back and narrating a stage that did not happen.Built to be honest rather than impressive
Reuse feedback is recorded with benefit
Unknown, and the transcript says "nothing moved — and that is the correct answer". Exposure alone moves no confidence score, and a demo that fabricated a human assessment to make the number rise would misrepresent the one property that makes this library worth trusting. The report names story 4.4 as where measurement happens and claims no model improvement anywhere.Stage 3 states a trust boundary the library cannot enforce. Which verification round is closed is the host's claim, not the library's finding — the aggregator cannot tell a genuine fix-and-rerun from a host parking a failure in a round it simply never closes. The sample stages the honest version and says so, rather than letting a reader infer a guarantee that is not there.
Run A drives the capture service directly, because
UseExperienceCapturerecords one MAF invocation as exactly one attempt (CaptureScope.cs:239,:285-311), so a multi-attempt run is not reachable through the middleware. The sample says this plainly instead of presenting hand-driven capture as the normal integration path. Logged as its own story.Verification reads the stores, not the narration
A golden transcript is checked in and compared byte-for-byte, and the tests read back the persisted record, the captured run's tool-call arguments, and the feedback ledger. That combination is what makes a stage that printed without running fail.
The secret tool argument is asserted absent from the render, the captured run, the persisted record, the injected block and the ledger row — and present-but-empty in the captured arguments, so a policy that dropped the field instead of redacting it would not pass either.
Determinism is asserted under invariant,
de-DEandtr-TR; the transcript emits\nexplicitly so it is byte-identical across platforms, not just across two runs on one box. CI now runs the sample twice and diffs both runs against the golden file.Review
Two reviewers. The mutation reviewer found 11 of 16 mutations survived the original 9 tests: deleting the reuse-feedback call entirely, moving the secret field to the allowed list (printing a live token beside a sentence claiming it was redacted), and making the failing attempt succeed all passed. The suite compared literal strings to literal strings. That is what the golden transcript and the store assertions replace.
Tests: 1096 → 1128, zero warnings under
TreatWarningsAsErrors.🤖 Generated with Claude Code