Skip to content

Close genkit 0.17 audit follow-ups - #1587

Merged
leehack merged 1 commit into
mainfrom
docs/genkit-1574-audit-followups
Oct 29, 2026
Merged

leehack merged 1 commit into
mainfrom
docs/genkit-1574-audit-followups

Conversation

@leehack

@leehack leehack commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Closes the #1574 audit follow-ups that cannot ride in the two-file genkit release prep (#1588). Merge this first.
  • New test: structured output with no declared tool parses no tool call. A user schema with a top-level response or tool_call field gets the reply exactly as the model wrote it, as the engine's final parse does (parseToolCalls only with tools). The grammar can produce this output, so the contract is reachable. It kills audit mutant m3 (parseToolCalls: true), which unwraps {"response":…} or turns {"tool_call":…} into a dropped call.
  • Migration guide: with thinking off, Qwen3.5 now returns <think>…</think>{...} as content, which fails JSON validation, as Seed-OSS already does.
  • genkit-package.md lower-bound lane: published llamadart 0.17.0 (the CI pin since Parse genkit structured output with ChatTemplateEngine.parseRendered #1574). CONTRIBUTING.md: llamadart 0.17.0 uses litert-lm-native v0.18.1-4 (liteRtLmReleaseTag at v0.17.0).

No new tests for mutants m6 and m8. Under the inputs model_action can reach they change nothing observable (probe of parseRendered on hosted 0.17.0 with the Qwen3.5, Seed-OSS, Qwen3-4B and generic renders, thinking on and off, with and without tools):

  • m6 (tools: null): tools reach the parse only on the generic format (the action rejects any other format with tools). There, a parse with and without tools gave identical results for every output tried, including undeclared names and mistyped or string-encoded arguments.
  • m8 (isPartial: true): with tools, the partial parse differs only on output that is not one complete JSON envelope. The unmutated envelope parse then rejects it with failedPrecondition anyway. Without tools, it differs only by trimming trailing whitespace from the visible JSON, and pinning that would assert nothing users rely on.

Production-readiness scope

  • User-facing scope: docs and a regression test only; no runtime change.
  • Supported platforms/paths: unchanged.
  • Unsupported or intentionally unavailable paths: unchanged.
  • Out of scope / follow-ups: after 3.6.0 is on pub.dev, update the published-pairing snippets in website/docs/integrations/genkit.md and choosing-an-api.md (3.5.0 / llamadart 0.16.1 today), as #1368 did for 3.5.0. They must name published versions, so the update cannot land before publication.

Completeness checklist

  • Declared scope is fully implemented, or explicitly reduced above.
  • Unsupported platform/option combinations fail loudly with actionable diagnostics or are clearly documented as unavailable. N/A: no runtime change.
  • Public API docs, README/website docs, examples, support matrices, and changelog entries are updated where relevant. The changelog wording is in the release prep, so this PR does not touch CHANGELOG.md.
  • Regression coverage covers the original issue plus key negative/version-skew paths where relevant.
  • Public capability changes update the validation coverage registry with a pack or a reasoned, issue-linked exemption; inventory checks pass (or N/A with scope). N/A: no capability change.
  • Security/privacy review completed: docs and a fake-runtime test only.
  • Follow-up work that is useful but not required for this PR is tracked in GitHub Issues, or explicitly marked "None" above. The post-publish pairing update is listed above.

PR type guidance

  • Feature PR: N/A, no feature.
  • Bugfix PR: N/A, test gap and docs. The new test fails under m3 and passes at head.
  • Docs-only PR: N/A, adds a test. No runtime behavior changes.

Test Plan

Pinned Flutter 3.47.1, macOS arm64, head ad876415a927f4688617c6b3a940109478e6d09e.

  • dart run tool/prepare_workspace.dart: no tracked changes.
  • dart format --output=none --set-exit-if-changed .: 0 changed.
  • dart analyze: no issues.
  • dart test -p vm -j 1 --exclude-tags local-only: N/A, no core change; dart test -p vm test/unit/tooling/ +1240 passed (~5 skipped).
  • dart test -p chrome --exclude-tags local-only: N/A, no core change.
  • Other targeted/local-only validation, from packages/genkit_llamadart against hosted llamadart 0.17.0:
    • dart format, dart analyze --fatal-infos, python3 .github/check_docs.py, .github unittest: pass.
    • dart test -x real-model: +468 passed.
    • stage_genkit_package.py to a fresh directory, then dart pub publish --dry-run: 0 warnings.
    • Mutant m3 applied to model_action.dart: the new test fails; restored: passes.
    • ./tool/docs/build_site.sh: pass.
    • verify_release_docs_versions.dart in a full clone: pass.

Matrix Evidence

Matrix row Scope covered Platform / model / backend Result Evidence / notes
genkit-contracts adapter tests, docs, archive macOS arm64, hosted llamadart 0.17.0, fake runtime PASS +468; dry-run 0 warnings
docs-site site build and links macOS arm64 PASS build_site.sh exit 0
release-doc-version-consistency install snippets full clone, 151 tags PASS core 0.17.0 and companions 0.17.0 verified
genkit-native-cpu native model contracts N/A N/A no runtime change

Review Notes

  • Independent review status: CLEAR at exact head ad87641 by a fresh auditor session (0 P1, 0 P2).
  • CI status / head SHA: pending, head ad876415a927f4688617c6b3a940109478e6d09e.

High-risk regression review

git diff --name-only --no-renames origin/main...HEAD | dart run tool/testing/classify_high_risk_changes.dart:

Classification: high-risk
Surfaces: regressionPolicy
Required matrix: dart run tool/testing/test_matrix.dart --tier high-risk

regressionPolicy comes from packages/genkit_llamadart/CONTRIBUTING.md; this PR changes one version reference in it.

  • Classification: high-risk (regressionPolicy)
  • Implementation task: this PR's author session
  • Independent blocking QA task: fresh auditor session, CLEAR, 0 known PR-caused P1, 0 unresolved threads
  • Exact head / current base: ad87641 / 9f1179a (merge-tree clean)
  • Production-branch deletion, bypass, or miswire proof: N/A, docs and a test only.
  • Affected-family real-model/artifact evidence: N/A, no runtime change. The Qwen3.5 note matches the Parse genkit structured output with ChatTemplateEngine.parseRendered #1574 audit's real-template probe.
  • Explicit unavailable-family or other N/A evidence: N/A.
  • Known PR-caused P1 regressions: 0
  • Unresolved review threads: 0

Pin that structured output without declared tools parses no tool call, so a
user schema with a top-level response or tool_call field is returned as the
model wrote it. Note the Qwen3.5 thinking-off parse change in the migration
guide, and move the lower-bound and LiteRT-LM references to llamadart 0.17.0.
@leehack leehack mentioned this pull request Oct 29, 2026
13 tasks done
@leehack
leehack marked this pull request as ready for review October 29, 2026 13:51
@leehack
leehack merged commit d6d0104 into main Oct 29, 2026
29 checks passed
@leehack
leehack deleted the docs/genkit-1574-audit-followups branch October 29, 2026 13:51
@github-actions

Copy link
Copy Markdown
Contributor

Chat app preview removed for leehack/llamadart-chat-pr-1587.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant