Add experimental NLI inference backend

[snellingio]

Summary

• Add an optional NLI backend behind the existing FastAPI contract.
• Batch candidate pairs and support shared-prefix cache reuse.
• Pin the tested checkpoint revision and report it in API responses.
• Keep the MLX backend as the default.
• Document backend settings, limits, and token counts.

Checks

• ruff format --check src tools tests
• ruff check src tools tests
• uv lock --check
• Server tests: 148 passed and 3 skipped.
• Full checkpoint tests on MPS: 3 passed.
• Python SDK tests: 4 passed.
• JavaScript SDK type check and tests passed.
• Final read-only review found no important problems.

Early results

One cache test reduced processed tokens from 612 to 222. Warm latency fell from 2.82 seconds to 1.60 seconds on MPS.

A small comparison matched 20 of 30 saved teacher answers. The split was 8 of 10 for Noul, 7 of 10 for Choice, and 5 of 10 for Score.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}