Skip to content

Close the #1837 LiteRT-LM test gaps - #1850

Closed
leehack wants to merge 1 commit into
mainfrom
test/1837-litert-gaps
Closed

leehack wants to merge 1 commit into
mainfrom
test/1837-litert-gaps

Conversation

@leehack

@leehack leehack commented Nov 9, 2026

Copy link
Copy Markdown
Owner

Summary

Production-readiness scope

  • User-facing scope: None; tests only.
  • Supported platforms/paths: VM tests on macOS and Linux.
  • Unsupported or intentionally unavailable paths: N/A.
  • Out of scope / follow-ups: None from this PR. The remaining 0.19.1 follow-ups from audits #1837 items (bundled-runtime marker export probe, coverage-lib row) are other lanes of #1845.

Completeness checklist

  • Declared scope is fully implemented, or explicitly reduced above.
  • Unsupported platform/option combinations fail loudly. (N/A, test-only)
  • Docs/changelog. (N/A, test-only)
  • Regression coverage covers the original issue plus key negative paths.
  • Validation coverage registry. (N/A)
  • Security/privacy review completed: no secrets or paths leak.
  • Follow-up work is tracked in GitHub Issues, or explicitly marked "None" above.

PR type guidance

  • Bugfix PR: N/A, test-only; no production change.

Test Plan

  • dart run tool/prepare_workspace.dart
  • dart format --output=none --set-exit-if-changed .
  • dart analyze
  • Touched suites (litert_lm_v1_embedding, litert_lm_gpu_adapter, litert_lm_service, litert_lm_embedding_native) plus test/unit/tooling/package_skills_test.dart and test/unit/test_structure: all pass (-j 1, isolated TMPDIR)
  • Full dart test -p vm -j 1 and chrome: N/A, CI runs them
  • coverage-lib: N/A, no lib/ change

Matrix Evidence

Matrix row Scope covered Platform / model / backend Result Evidence / notes
N/A Test-only LiteRT-LM fixtures macOS arm64 VM, compiled C fixtures Pass mutations below

Mutation results (each reverted):

# Mutation Result
1 LiteRtLmRuntimeClient._selectRuntime passes marker 1 when _facadeLibraryPath == null (env-dir route) new env-dir refusal tests fail (3 failures: linux_arm64, linux_x64, windows_x64)
2 _listDevices loops over the allocated count instead of the rewritten count.value reads the count the second call rewrites fails
3 litert_lm_service.dart GPU rejection uses Platform.operatingSystem instead of liteRtLmGateAbi() 3 fails to load with a typed error tests fail
4 N/A (flake fix) see below

Item 1: new test/fixtures/litert_lm_v1_env_dir_embedding.dart runs the default LiteRtLmRuntimeClient() in a child process with LLAMADART_LITERT_LM_LIB_DIR set to a directory holding the compiled unmarked fixture runtime plus stub companions, with the gate ABI overridden to Linux/Windows; it must refuse with the marker message. A marked runtime through the same route is admitted (control).

Item 4: LiteRtLmBackend.embeddingTimeout is a constructor setting with no per-call override, so the reuse call cannot get its own timeout without a lib/ change. The deadline test now uses a 1 s deadline instead of 100 ms (the assertion is unchanged), which gives the reuse call 10x the headroom. Stability: 20/20 passes under CPU load (3x core count busy loops). The original 100 ms version also passed 20/20 under the same load, so the flake was not reproduced locally; the change is a risk reduction, not a demonstrated fix.

Review Notes

  • Independent review status: pending
  • CI status / head SHA: pending

High-risk regression review

  • Classification: standard
  • Implementation task: N/A
  • Independent blocking QA task: N/A
  • Exact head / current base: N/A
  • Production-branch deletion, bypass, or miswire proof: N/A
  • Affected-family real-model/artifact evidence: N/A
  • Explicit unavailable-family or other N/A evidence: N/A
  • Known PR-caused P1 regressions: 0
  • Unresolved review threads: 0

Refs #1837 #1845

@leehack

leehack commented Nov 9, 2026

Copy link
Copy Markdown
Owner Author

Landed through the 0.19.2 batch integration PR #1870 (rebase merge, one commit per constituent; this PR's pinned head was audited/CI-gated there).

@leehack leehack closed this Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Chat app preview removed for leehack/llamadart-chat-pr-1850.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant