Repository navigation
feat(llma): add typed offline evaluations SDK - #1005
Conversation
posthog-python Compliance ReportDate: 2026-10-02T17:09:24.431900+00:00 ✅ All Tests Passed!121/121 tests passed Capture_V1 Tests✅ 95/95 tests passed View Details
Capture_Ai Tests✅ 5/5 tests passed View Details
Feature_Flags Tests✅ 17/17 tests passed View Details
Feature_Flags_Local_Evaluation Tests✅ 4/4 tests passed View Details
|
|
[Medium risk] Adds a new offline evaluations SDK module. The PR appears safe to merge; the remaining unchecked scorer-version return type is non-blocking. Reviews (2) · Last reviewed commit: "fix(llma): reject integer scores that lo..." |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| def create_version( | ||
| self, | ||
| scorer_id: str | UUID, | ||
| *, | ||
| config: _ConfigT, | ||
| base_version: int | None = None, |
There was a problem hiding this comment.
create_version() infers its return type from the supplied config without checking the target scorer’s kind. Passing a numeric config for a boolean scorer can leave callers with a value typed as Scorer[NumericScorerConfig] even if the operation is rejected or returns a boolean scorer. The async method has the same issue. Tie the config type to the scorer kind instead of relying on a cast.
Prompt To Fix With AI
This is a comment left during a code review.
Path: posthog/ai/evaluations/_scorers.py
Line: 367-372
Comment:
**Version return type is unchecked** `create_version()` infers its return type from the supplied config without checking the target scorer’s kind. Passing a numeric config for a boolean scorer can leave callers with a value typed as `Scorer[NumericScorerConfig]` even if the operation is rejected or returns a boolean scorer. The async method has the same issue. Tie the config type to the scorer kind instead of relying on a cast.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
marandaneto
left a comment
marandaneto
left a comment
There was a problem hiding this comment.
The code looks good; no qualifying findings. Verdict: correct.
|
tagging @dustinbyrne for a 2nd look |
marandaneto
left a comment
marandaneto
left a comment
There was a problem hiding this comment.
Five non-blocking public API ergonomics suggestions, posted at the user's request. No new blocking findings; these do not change the earlier correctness verdict.
marandaneto
left a comment
marandaneto
left a comment
There was a problem hiding this comment.
Five additional suggestions for future public API expansion, with illustrative examples of possible future usage. These can be considered here or in follow-up work.
| ScoreValue = bool | int | float | list[str] | ||
|
|
||
|
|
||
| class ExperimentOptions(TypedDict, total=False): |
There was a problem hiding this comment.
suggestion: Leave room for experiment metadata — it would be useful to attach JSON metadata such as temperature, retrieval settings, or judge-prompt versions to a run, rather than encoding everything in its name or fitting it into the fixed version fields. An optional typed metadata field could be a small starting point, with descriptions/tags added later if needed. This can be a follow-up coordinated with the backend, preserving metadata in the original submission for exact replay. Illustrative future usage, not currently supported:
experiment = evaluations.create_experiment(name="baseline", metadata={"temperature": 0.2, "retrieval_strategy": "hybrid"})Reproduction: not reproduced — inspected ExperimentOptions; this proposes an additional backend/SDK field, so the example cannot run against the current API.
| self.receipt = receipt | ||
| return receipt | ||
|
|
||
| def complete(self) -> ExperimentReceipt: |
There was a problem hiding this comment.
suggestion: Leave room for experiment-level summaries — metrics such as precision, recall, total cost, or latency percentiles describe an entire run rather than one item/scorer pair. It would be useful to accommodate a separate summary upload API later, without inventing synthetic items or changing accepted-item/result counts. The implementation can come in a follow-up once the backend contract, value types, replay identity, and relationship to versioned scorers are agreed. Illustrative future usage, not currently supported:
experiment.upload_summary(metrics={"precision": 0.91, "p95_latency_ms": 320})
experiment.complete()Reproduction: not reproduced — inspected the item/result upload and completion surface; this is an experiment-level API proposal without a current endpoint to exercise.
|
|
||
| def __init__(self, transport: SyncTransport, project_id: int) -> None: | ||
| self._transport = transport | ||
| self._base = f"/api/projects/{project_id}/llm_analytics/score_definitions/" |
There was a problem hiding this comment.
this ships the legacy llm_analytics/ prefix in released sdk versions, so the server would need an alias to ever move to ai_observability/. probably worth deciding on purpose before release, wdyt?
3 participants
💡 Motivation and Context
Add
posthog.ai.evaluationsso Python callers can upload offline evaluation results and manage experiments through typed synchronous and asynchronous clients. Includes acknowledged bulk uploads, stable identities for safe retries and recovery, and explicit completion/failure.Scorer management includes typed boolean, numeric, and categorical configurations with polarity, metadata updates, immutable version creation, and programmatic version-ID discovery. Includes usage examples and a minor changeset.
💚 How did you test it?
📝 Checklist
If releasing new changes
.sampo/changesets/.🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Implemented with Codex and collaborating agents using shell/Python tooling, Git, and the GitHub CLI. Local Codex session.
The API follows the current backend contract. Configuration types expose supported fields; scorer mutations avoid automatic retries, while replay-safe offline operations retain identities and content.