Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,25 @@
# Changelog

## 0.6.0

New `LocalVerifier` (`pip install "cacheverifier[local]"`, Managed plans):
scores gray-zone hits on your own machines with the model CacheVerifier
fine-tuned for you, deciding exactly as the hosted `/v1/verify` would. It
downloads the live model, checks it against reference verdicts from the
service before using it, switches to newer models as they go live, and, while
auditing is on, uploads each decision without the texts so audits and risk
monitoring keep working. Falls back to the hosted API until a model is ready.

`CacheVerifier.verify()` takes `similarity_score`, `cached_query` and
`runner_up_similarity_score` (all optional), which a fine-tuned model's joint
rules use; before this the client couldn't send them. `feedback()` takes
`cached_query` and `verification_id`. `VerifyResult` gains `decision_rule`,
`audit` and `verification_id`, which the API already returned.

The offline Health Check now truncates long inputs exactly as the service
does since its verifier-core 0.7.0: a long query next to a short answer keeps
the room the answer doesn't need, instead of being cut to half the budget.

## 0.5.0

New LangChain integration: `cacheverifier.integrations.langchain.VerifiedCache`
Expand Down
40 changes: 39 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ pip install cacheverifier
pip install "cacheverifier[gptcache]"
# with the offline Health Check (adds torch + sentence-transformers):
pip install "cacheverifier[healthcheck]"
# to score with your fine-tuned model on your own machines (same deps):
pip install "cacheverifier[local]"
```

Requires Python 3.9+. The only runtime dependency is `httpx` — the extras above
Expand Down Expand Up @@ -163,6 +165,40 @@ if job.get("result_model_version"):
`cv.dry_run([...])` reports the same baseline-vs-tuned AUC on examples you pass directly,
without writing anything or deploying a model.

## Local scoring (Managed plans)

`LocalVerifier` runs the model CacheVerifier fine-tuned for you on your own
machines: no network round trip on your request path, and your query and
answer texts stay with you. Fine-tuning, drift monitoring, audits and
certification keep running on the service.

```python
from cacheverifier import LocalVerifier

cv = LocalVerifier(api_key="cv_...") # downloads your live model in the background
r = cv.verify(query, cached_answer, similarity_score=0.87, cached_query=cached_question)
if r.approved:
... # serve from cache
else:
answer = call_llm(query)
if r.audit: # a randomized audit: report whether the cached answer was right
cv.feedback(query, cached_answer, was_correct=same_answer(answer, cached_answer), result=r)
```

- It decides exactly as the hosted `/v1/verify` would with that model: every download
carries reference verdicts from the service, and a model that doesn't reproduce them
locally is not used.
- When a newer model goes live (after a fine-tune or a scheduled recalibration), it is
downloaded, checked, and switched to without a restart.
- While auditing is on, each decision is uploaded as scores, verdicts and a keyed hash of
the pair, never the texts, so audits and `/v1/monitor/risk` work as with the hosted API.
- Until a model is ready, calls go to the hosted API (`fallback="hosted"`, the default)
or are rejected (`fallback="reject"`). Pass `wait=True` to block until the first
download finishes (a model is ~330MB).
- The audit draw happens in this client. The risk bound assumes it is independent of
whether the answer is right, which the client guarantees unless it is modified.
- Not supported yet: the online ACI threshold (local scoring uses the model's fixed one).

## Local Health Check (offline)

`cv.dry_run()` still uploads your examples to the API. If that's a blocker — a
Expand Down Expand Up @@ -224,7 +260,7 @@ verdict

| method | endpoint |
|---|---|
| `verify(query, candidate_answer)` | `POST /v1/verify` |
| `verify(query, candidate_answer, similarity_score=..., cached_query=...)` | `POST /v1/verify` |
| `verify_batch(pairs)` | `POST /v1/verify/batch` |
| `feedback(...)` / `feedback_batch(items)` | `POST /v1/feedback` / `/batch` |
| `finetune(...)` / `dry_run(examples, ...)` | `POST /v1/finetune/jobs` / `/dry-run` |
Expand All @@ -233,6 +269,8 @@ verdict
| `drift_status()` | `GET /v1/monitor/drift-status` |
| `gray_zone_threshold()` | `GET /v1/monitor/gray-zone-threshold` |
| `usage()` / `savings()` | `GET /v1/usage/status` / `/savings` |
| `LocalVerifier.verify(...)` / `verify_batch(...)` | local; decisions to `POST /v1/verify/decisions` |
| `LocalVerifier.refresh()` | `GET /v1/models/active` + `/v1/models/{id}/files/{name}` |

Non-2xx responses raise `CacheVerifierError` (`.status_code`, `.detail`).

Expand Down
13 changes: 11 additions & 2 deletions cacheverifier/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,15 @@
"""

from cacheverifier.client import CacheVerifier, CacheVerifierError, ShadowVerdict, VerifyResult
from cacheverifier.local import LocalVerifier, ModelCheckFailed

__version__ = "0.5.0"
__all__ = ["CacheVerifier", "CacheVerifierError", "ShadowVerdict", "VerifyResult", "__version__"]
__version__ = "0.6.0"
__all__ = [
"CacheVerifier",
"CacheVerifierError",
"LocalVerifier",
"ModelCheckFailed",
"ShadowVerdict",
"VerifyResult",
"__version__",
]
41 changes: 6 additions & 35 deletions cacheverifier/_healthcheck/_finetune.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
from torch.utils.data import DataLoader

from cacheverifier._healthcheck._base_models import DEFAULT_BASE_MODEL
from cacheverifier._inference import MAX_SEQUENCE_LENGTH, smart_truncate_pair

# The four third-party imports above are from the `healthcheck` extra. This module is
# only ever imported from inside `cacheverifier._healthcheck.cli.run()`,
Expand All @@ -36,13 +37,11 @@
"""`cross-encoder/nli-*` models' class order, used when the loaded model
doesn't expose an `id2label` naming them."""

MAX_SEQUENCE_LENGTH = 128
"""Token cap applied identically at train and score time. Matches the
hosted service; BERT-style attention memory is O(n^2) in sequence length,
and this keeps a CPU run well-behaved."""

_CONTENT_TOKEN_BUDGET = MAX_SEQUENCE_LENGTH - 3 # [CLS] + 2x [SEP]
_MIN_QUERY_TOKENS = 32
# Token cap and truncation shared with local scoring, so a Health Check
# trains with exactly what the service and `LocalVerifier` score with. Before
# 0.6.0 this module kept its own copy of the truncation, which had fallen
# behind the service's 0.7.0 change (a long query next to a short answer
# was cut to half the budget).

TRAIN_BATCH_SIZE = 4
TRAIN_EPOCHS = 3
Expand Down Expand Up @@ -102,34 +101,6 @@ def auc_delta(self) -> float:
return self.auc_tuned - self.auc_baseline


def _head_tail_truncate(ids: list[int], budget: int, head_ratio: float = 0.6) -> list[int]:
if len(ids) <= budget:
return ids
head_n = int(budget * head_ratio)
tail_n = budget - head_n
return ids[:head_n] + (ids[-tail_n:] if tail_n > 0 else [])


def smart_truncate_pair(query: str, candidate_answer: str, tokenizer: Any) -> tuple[str, str]:
"""Tokenizer-aware head+tail truncation, applied identically at train
and score time (a model trained on one truncation policy and scored
with another sees a different token window than it was optimized
against). A no-op whenever the pair already fits."""
query_ids = tokenizer.encode(query, add_special_tokens=False)
answer_ids = tokenizer.encode(candidate_answer, add_special_tokens=False)
if len(query_ids) + len(answer_ids) <= _CONTENT_TOKEN_BUDGET:
return query, candidate_answer

query_budget = min(len(query_ids), max(_MIN_QUERY_TOKENS, _CONTENT_TOKEN_BUDGET // 2))
answer_budget = _CONTENT_TOKEN_BUDGET - query_budget
query_ids = _head_tail_truncate(query_ids, query_budget)
answer_ids = _head_tail_truncate(answer_ids, answer_budget)
return (
tokenizer.decode(query_ids, skip_special_tokens=True),
tokenizer.decode(answer_ids, skip_special_tokens=True),
)


def roc_auc(scores: np.ndarray, labels: np.ndarray) -> float:
"""Rank-based AUC, no sklearn dependency. NaN if either class is absent."""
order = np.argsort(scores)
Expand Down
162 changes: 162 additions & 0 deletions cacheverifier/_inference.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
"""The scoring and decision steps the hosted service applies to one
(query, candidate_answer) pair, for running a downloaded model locally.

A port of the hosted service's inference path: input composition with the
`cached_query` diff, tokenizer-aware truncation, and the threshold / joint
decision rules. Training, certification and monitoring stay on the service.
Pure Python (the tokenizer is passed in), so it imports without the `local`
extra.

Kept in step with the service by two checks: the service's own test suite
compares these functions with its implementation, and every model download
carries reference verdicts that `LocalVerifier` reproduces before it decides
anything (see `cacheverifier.local`).
"""

from __future__ import annotations

import difflib
import math
from dataclasses import dataclass
from typing import Any

MAX_SEQUENCE_LENGTH = 128
"""Token cap applied at train and score time, as on the service."""

CONTENT_TOKEN_BUDGET = MAX_SEQUENCE_LENGTH - 3 # [CLS] + 2x [SEP]
MIN_QUERY_TOKENS = 32
QUERY_DIFF_MAX_WORDS = 12


def query_diff_summary(current_query: str, cached_query: str) -> str:
"""Word-level diff of the cache entry's own query against the request's,
e.g. `removed: pause | added: cancel`."""
old, new = cached_query.split(), current_query.split()
sm = difflib.SequenceMatcher(a=old, b=new, autojunk=False)
removed: list[str] = []
added: list[str] = []
for tag, i1, i2, j1, j2 in sm.get_opcodes():
if tag in ("replace", "delete"):
removed.extend(old[i1:i2])
if tag in ("replace", "insert"):
added.extend(new[j1:j2])
parts = []
if removed:
parts.append("removed: " + " ".join(removed[:QUERY_DIFF_MAX_WORDS]))
if added:
parts.append("added: " + " ".join(added[:QUERY_DIFF_MAX_WORDS]))
return " | ".join(parts) if parts else "no change"


def compose_query_input(current_query: str, cached_query: str | None) -> str:
if not cached_query:
return current_query
return f"{current_query} [qdiff] {query_diff_summary(current_query, cached_query)}"


def _head_tail_truncate(ids: list[int], budget: int, head_ratio: float = 0.6) -> list[int]:
if len(ids) <= budget:
return ids
head_n = int(budget * head_ratio)
tail_n = budget - head_n
return ids[:head_n] + (ids[-tail_n:] if tail_n > 0 else [])


def smart_truncate_pair(query: str, candidate_answer: str, tokenizer: Any) -> tuple[str, str]:
"""Tokenizer-aware head+tail truncation, a no-op whenever the pair fits.
The answer gets at most half the budget while the query wants more; the
query gets whatever the answer doesn't need."""
query_ids = tokenizer.encode(query, add_special_tokens=False)
answer_ids = tokenizer.encode(candidate_answer, add_special_tokens=False)
if len(query_ids) + len(answer_ids) <= CONTENT_TOKEN_BUDGET:
return query, candidate_answer
half = CONTENT_TOKEN_BUDGET // 2
query_budget = min(len(query_ids), max(MIN_QUERY_TOKENS, half, CONTENT_TOKEN_BUDGET - len(answer_ids)))
answer_budget = CONTENT_TOKEN_BUDGET - query_budget
query_ids = _head_tail_truncate(query_ids, query_budget)
answer_ids = _head_tail_truncate(answer_ids, answer_budget)
return (
tokenizer.decode(query_ids, skip_special_tokens=True),
tokenizer.decode(answer_ids, skip_special_tokens=True),
)


def _sigmoid(x: float) -> float:
if x >= 0.0:
return 1.0 / (1.0 + math.exp(-x))
e = math.exp(x)
return e / (1.0 + e)


@dataclass(frozen=True)
class JointRule:
"""A fitted joint decision rule, as the service stores it: approve iff
the verifier score reaches `verifier_floor` (when set) and the fused
probability over the standardized similarity, verifier score (and, with
`uses_gap`, the gap to the runner-up match) reaches `threshold`."""

coef_similarity: float
coef_verifier: float
intercept: float
similarity_mean: float
similarity_std: float
verifier_mean: float
verifier_std: float
threshold: float
coef_gap: float = 0.0
gap_mean: float = 0.0
gap_std: float = 1.0
uses_gap: bool = False
verifier_floor: float | None = None

def approves(
self, verifier_score: float, similarity_score: float, runner_up_similarity_score: float | None = None
) -> bool:
if self.verifier_floor is not None and verifier_score < self.verifier_floor:
return False
z_sim = (similarity_score - self.similarity_mean) / self.similarity_std
z_ver = (verifier_score - self.verifier_mean) / self.verifier_std
logit = self.coef_similarity * z_sim + self.coef_verifier * z_ver + self.intercept
if self.uses_gap:
if runner_up_similarity_score is None:
raise ValueError("this joint rule uses the similarity gap; runner_up_similarity_score is required")
z_gap = ((similarity_score - runner_up_similarity_score) - self.gap_mean) / self.gap_std
logit += self.coef_gap * z_gap
return _sigmoid(logit) >= self.threshold

@classmethod
def from_dict(cls, d: dict[str, Any]) -> JointRule:
floor = d.get("verifier_floor")
return cls(
coef_similarity=float(d["coef_similarity"]),
coef_verifier=float(d["coef_verifier"]),
intercept=float(d["intercept"]),
similarity_mean=float(d["similarity_mean"]),
similarity_std=float(d["similarity_std"]),
verifier_mean=float(d["verifier_mean"]),
verifier_std=float(d["verifier_std"]),
threshold=float(d["threshold"]),
coef_gap=float(d.get("coef_gap", 0.0)),
gap_mean=float(d.get("gap_mean", 0.0)),
gap_std=float(d.get("gap_std", 1.0)),
uses_gap=bool(d.get("uses_gap", False)),
verifier_floor=float(floor) if floor is not None else None,
)


def decide(
score: float,
threshold: float,
joint_rule: JointRule | None,
joint_rule_gap: JointRule | None,
similarity_score: float | None,
runner_up_similarity_score: float | None,
) -> tuple[bool, str]:
"""(approved, decision_rule), with the service's precedence: the gap
rule when both similarity scores are given, then the 2-feature rule
when the similarity is, else the plain threshold."""
if joint_rule_gap is not None and similarity_score is not None and runner_up_similarity_score is not None:
return joint_rule_gap.approves(score, similarity_score, runner_up_similarity_score), "joint_gap"
if joint_rule is not None and similarity_score is not None:
return joint_rule.approves(score, similarity_score), "joint"
return score >= threshold, "threshold"
Loading
Loading