feat(aio): publish tagged offline eval results to posthog by Radu-Raicea · Pull Request #119849 · PostHog/posthog

Radu-Raicea · 2026-10-19T19:16:20Z

Problem

Developers cannot compare most PostHog AI and Slack offline eval results in PostHog, and existing runs lack consistent team and component tags.

Changes

Publish public PostHog AI and Slack app suites to PostHog alongside Braintrust.
Publish the 11 experiment skill suites only to PostHog through an explicit reporting override.
Apply team and component tags to experiment runs, including the existing SQL and Signals publishers.
Pin tagged scorer versions in shared catalogs; suite enrollment changes are mechanical.
Keep completed uploads successful when tagging fails, and avoid repeating mutable tag updates after interrupted delivery.
Upgrade posthoganalytics to 7.73.0 for the tag methods added in PostHog/posthog-python#1085.

Note

Uploading and tagging require a personal key with offline_evaluation_ingestion:write and evaluation:write.

How did you test this code?

Test rationale: Existing publishing and enrollment tests now cover independent destinations and complete scorer mappings. HTTP-boundary tests cover tag failures and interrupted updates.

Automated reporting tests use the released SDK. Harness discovery and repository-wide type checking cover the updated dependency.

Full live agent eval runs were not executed.

Release status

No feature flag controls this change

Docs update

Updated the existing offline evaluation reporting guide and harness README with tags, required scopes, and PostHog-only reporting.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Codex, GPT-6

Tools: Git, GitHub and PostHog connectors, repository validation.
Skills: /writing-evals, /writing-tests, /writing-dataclasses, /writing-code-comments, /establishing-code-ownership, /posthog-efficient-validation, /running-ci-preflight, /writing-pr-descriptions, create-pr, github-efficient-operations, llma-git-workflow, posthog:posthog, posthog:analyzing-offline-evaluations.
The scope expanded to include PostHog-only reporting for the experiment suites. Existing prompts and cases remain unchanged.
The duplicate search found no overlapping reporting PR. No customer conversations, trace payloads, or credentials appear in the patch.

trunk-io · 2026-10-19T19:16:25Z

😎 Merged successfully - details.

greptile-apps

Your trial has ended. Reactivate Greptile to resume code reviews.

github-actions · 2026-10-19T19:18:25Z

🤖 CI report

⚠️

Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

⚠️

Duplication (Python) — 1 new duplicated block (worst 82 tokens)

New Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

First copy	Second copy	Lines	Tokens
`products/posthog_ai/evals/product_analytics/eval_sql_shaped_routing.py:11`	`products/posthog_ai/evals/product_analytics/eval_typed_runner_lock_in.py:27`	11	82

✅ Duplication (TypeScript) — clean

New TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

✅ Playwright — all passed

All tests passed.

View test results →

✅ Hobby preview — passed

Hobby deployment smoke test passed successfully.

_{Run 39431834720}

posthog

PostHog Review · Standard · reviewed 6717d45: 1 Must fix, 1 Should fix.

This is the only automatic review while this pull request is a draft. Comment @posthog review to review new changes sooner. Otherwise, a review of the new changes runs when you mark it ready for review.

parameterai · 2026-10-19T19:30:40Z

Risk: No findings

Since the last review, the only change is a single test-only line in test_offline_delivery.py that re-enables the offline_results logger during an interrupted-tagging test so captured warnings are asserted. No production code, credentials, network calls, or access control paths were altered, so no new security risk is introduced.

_{Sentinel reviewed b018958 · Review settings}

greptile-apps

Your trial has ended. Reactivate Greptile to resume code reviews.

pr-assigner-resolver-posthog · 2026-10-19T19:31:11Z

👀 Auto-assigned reviewers

These soft owners were skipped because they only have minor changes here. Nothing blocks merge, so self-assign if you'd like a look:

@PostHog/team-self-driving (products/signals/product.yaml)

Soft owners come from each directory's owners.yaml and each product's product.yaml (resolved nearest-file-wins). For a skipped owner, the locator is the file that decided it. Generated files and lockfiles are ignored when deciding ownership.

trunk-io · 2026-10-19T19:53:24Z

_{View Full Report ↗︎ ⋅ Docs}

bernatixer

Key scopes in the docs. The key table (docs/internal/ai-offline-evaluation-reporting.md:52) and the README setup line (products/posthog_ai/eval_harness/README.md:403) still say offline_evaluation_ingestion:write is enough. Since tagging needs evaluation:write (doc line 98), anyone following them gets a tag warning on every run and no tags. Worth listing both scopes there.
Broken anchor. products/posthog_ai/eval_harness/README.md:406 still links #sql-offline-experiment-pilot, which went away with the heading rename. It should now be #configured-offline-experiment-suites.
writing-evals skill. .agents/skills/writing-evals/SKILL.md:51 still says SandboxedPrivateEval uploads to neither service and that PostHog uploads follow no_send_logs. Both stop being true with send_to_posthog=True. It also has no offline_suite(...) step, so a new suite written from it won't publish, while the doc now says all public suites do.

VojtechBartos

🚀

…l-tags

posthog

PostHog Review · Standard · reviewed 5440696: 1 Should fix.

posthog · 2026-10-20T13:06:28Z

+        suite
+        for suite in discover_suites()
+        if suite.fn.__module__.startswith(("products.posthog_ai.evals.", "products.slack_app.evals."))
+        and hasattr(importlib.import_module(suite.fn.__module__), "offline_suite")


P2 · Enroll the remaining public product analytics suites

When running eval_chart_not_table, eval_typed_runner_misses, or eval_justified_sql_controls, results still publish only to Braintrust because those public suites lack offline_suite. This filter silently excludes their modules from enrollment validation, leaving the reporting gap undetected. Enroll these suites and validate coverage without requiring an existing catalog import.

deployment-status-posthog · 2026-10-20T15:33:35Z

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-10-20 15:33 UTC	Run
prod-us	✅ Deployed	2026-10-20 15:47 UTC	Run
prod-eu	✅ Deployed	2026-10-20 15:48 UTC	Run

Radu-Raicea added 2 commits October 19, 2026 15:11

feat(aio): publish tagged offline eval results to posthog

c892c3d

chore(aio): merge master into offline eval reporting

6717d45

Radu-Raicea self-assigned this Oct 19, 2026

greptile-apps Bot reviewed Oct 19, 2026

View reviewed changes

Radu-Raicea added the skip-inkeep-docs Use this label to skip an Inkeep docs PR in posthog.com label Oct 19, 2026 — with ChatGPT Codex Connector

posthog Bot reviewed Oct 19, 2026

View reviewed changes

Comment thread products/posthog_ai/eval_harness/offline_results.py Outdated

Comment thread products/posthog_ai/evals/offline_config.py Outdated

Radu-Raicea added 2 commits October 19, 2026 15:24

chore(aio): upgrade python sdk for offline evaluation tags

225052a

chore(aio): merge master into offline eval reporting

d4b7f33

Radu-Raicea marked this pull request as ready for review October 19, 2026 19:30

greptile-apps Bot reviewed Oct 19, 2026

View reviewed changes

pr-assigner-resolver-posthog Bot requested a review from a team October 19, 2026 19:31

Radu-Raicea requested review from a team and VojtechBartos October 19, 2026 19:32

fix(aio): omit sandbox exit scores from one-shot evals

3924aa5

bernatixer approved these changes Oct 19, 2026

View reviewed changes

Comment thread products/posthog_ai/eval_harness/offline_results.py Outdated

Comment thread products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py Outdated

VojtechBartos approved these changes Oct 20, 2026

View reviewed changes

Radu-Raicea added 2 commits October 20, 2026 08:57

Merge remote-tracking branch 'origin/master' into feat/ai-offline-eva…

0d63ef5

…l-tags

fix(aio): preserve completed eval uploads during tagging

5440696

posthog Bot reviewed Oct 20, 2026

View reviewed changes

chore(aio): enable log capture in offline delivery tests

b018958

trunk-io Bot merged commit e341b72 into master Oct 20, 2026
245 checks passed

trunk-io Bot deleted the feat/ai-offline-eval-tags branch October 20, 2026 14:59

Conversation

Radu-Raicea commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Changes

How did you test this code?

Release status

Docs update

🤖 Agent context

Uh oh!

trunk-io Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 CI report

Uh oh!

posthog Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

parameterai Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

Uh oh!

pr-assigner-resolver-posthog Bot commented Oct 19, 2026

👀 Auto-assigned reviewers

Uh oh!

trunk-io Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bernatixer left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

VojtechBartos left a comment

Choose a reason for hiding this comment

Uh oh!

posthog Bot left a comment

Choose a reason for hiding this comment

Uh oh!

posthog Bot Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

deployment-status-posthog Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploy status

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Radu-Raicea commented Oct 19, 2026 •

edited

Loading

trunk-io Bot commented Oct 19, 2026 •

edited

Loading

github-actions Bot commented Oct 19, 2026 •

edited

Loading

parameterai Bot commented Oct 19, 2026 •

edited

Loading

trunk-io Bot commented Oct 19, 2026 •

edited

Loading

bernatixer left a comment •

edited

Loading

deployment-status-posthog Bot commented Oct 20, 2026 •

edited

Loading