Repository navigation
0.6.0: LocalVerifier (score locally with your fine-tuned model) + hosted verify() sends similarity/cached_query - #7
Merged
Conversation
…y() sends similarity and cached_query; healthcheck truncates like the service
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
This is the client side of local scoring for Managed plans; the service side is cacheverifier-service #128 and #129. A tenant scores gray-zone hits on its own machines with the model the service fine-tuned for it. Fine-tuning, drift monitoring, audits and certification keep running on the service.
Changes
cacheverifier.local.LocalVerifier, also exported from the package root:GET /v1/models/activeand/v1/models/{id}/files/{name}. An interrupted download resumes with a Range request, and every file is checked against its sha256. It then loads the model withsentence-transformers(newlocalextra, the same stack ashealthcheck).ModelCheckFailed).VerifyResultas the hostedverify(). It decides with the threshold, joint and gap rules (floor included), with the service's precedence.audit_rate > 0it draws audits locally and queues each decision: score, rule, verdicts, audit probability, and an HMAC of the pair under the tenant'spair_key, never the texts. The queue is uploaded toPOST /v1/verify/decisionsevery 30 seconds, or sooner when 1,000 have built up, and holds at most 100k decisions during an outage.feedback(..., result=r)uploads a still-pending decision first, then links the feedback with itsverification_id.fallback="hosted"(default) sends calls to the hosted API until a model is ready;"reject"returns a degraded rejection.wait=Trueblocks until the first model is ready.cacheverifier._inference: a pure-Python port of the service's inference path, namelycompose_query_inputwith thecached_querydiff,smart_truncate_pair,JointRuleanddecide. The healthcheck now imports its truncation from here.verify()takessimilarity_score,cached_queryandrunner_up_similarity_score. Before this, the client could not send them, so a fine-tuned model's joint rules never applied to SDK calls.feedback()takescached_queryandverification_id.VerifyResultgainsdecision_rule,auditandverification_id.Testing
tests/test_local.py: 9 offline tests, using a mock service and a word-overlap model.verifier-coreandapp.local_bundle.REFERENCE_PAIRSagainst this client:cached_query, the check passes, the scores differ by at most 1.4e-06, and all 20 verdicts are equal, covering all three rules;cached_querycomposition matches.ruffandmypyare clean. On Python 3.9 (Docker),test_localandtest_clientpass.Release
Don't release until the service side (#128, #129) is deployed. It is now: #128 is live and #129 is waiting to merge.