Skip to content

feat(flags): report unresolved local flags on EvaluateFlagsAsync snapshots - #357

Merged
gustavohstrassburger merged 5 commits into
mainfrom
posthog/unresolved-local-flags
Oct 29, 2026
Merged

gustavohstrassburger merged 5 commits into
mainfrom
posthog/unresolved-local-flags

Conversation

@gustavohstrassburger

@gustavohstrassburger gustavohstrassburger commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

💡 Motivation and Context

Why: sdk-specs #85 defines how a server SDK reports flags that local evaluation could not resolve. Today such a flag is absent from the EvaluateFlagsAsync snapshot, so it looks the same as a flag that does not exist, and $feature_flag_called reports flag_missing for both. With OnlyEvaluateLocally this is permanent. posthog-node already implements this (PostHog/posthog-js#5234).

Changes:

  • FeatureFlagEvaluations.UnresolvedFlags maps each unresolved flag to an UnresolvedFlagReason (ExperienceContinuity, UnsupportedDefinition, UnresolvedDependency, MissingContext; ToValueString() and the JSON form give the spec strings). When several causes apply, the spec's fixed precedence picks one. Only, OnlyAccessed and Empty snapshots have no entries, and reading the map is not an access.
  • A snapshot read of an unresolved flag reports local_evaluation_inconclusive instead of flag_missing.
  • InconclusiveMatchException and RequiresServerEvaluationException carry an internal reason, so their shipped public surface does not change. The date and semver parsers tell a bad filter value (UnsupportedDefinition) from a bad person value (MissingContext).
  • LocalFeatureFlagsLoader logs a warning for flags local evaluation can never resolve, deduplicated per flag on the definition plus the reason and rebuilt on each 200. A 304 does not repeat it. The per-evaluation static-cohort error log is removed.

Behavior changes (in the changeset):

  • A condition no longer stops at the first filter it cannot check. If another filter does not match, the condition resolves locally as no match. A static cohort still forces the fallback only when it is the first filter in its condition that cannot be checked, as in posthog-node.
  • An inactive flag with experience continuity resolves to false locally.

Divergences from the spec or posthog-node:

  • A missing group key still resolves to false and there is no device-id bucketing, so neither gives MissingContext and the device-id acceptance row does not apply.
  • The definition model has no version field, so the warning fingerprint compares the parsed definition. There is no shared definitions cache, so the shared-cache warning row does not apply.
  • OnlyAccessed() with nothing accessed still returns all flags (existing behavior), so the spec step "only accessed returns no flags" differs. Its unresolved map is empty.

💚 How did you test it?

About 100 new xUnit tests mirror every scenario tagged @unresolved_flags_capable and @unresolvable_flag_warning_capable in acceptance/public/evaluate-flags.feature, plus precedence, the condition loop, dependencies, warning dedup and pruning, and unexpected errors. Run locally with the .NET 9 SDK:

  • dotnet restore --locked-mode
  • dotnet build --configuration Release --no-restore (0 warnings, 0 errors, PublicApiAnalyzers included)
  • dotnet test --configuration Release --no-build -f net8.0 (UnitTests 1504 passed, 2 skipped; AspNetCore 68; AI 19)
  • bin/fmt --check

The netcoreapp3.1 tests did not run locally because the machine has no libssl 1.1; CI covers them. No manual testing in an app.

📝 Checklist

  • All commits are signed (see Commit signing)
  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran pnpm changeset to generate a changeset file

🤖 Agent context

Autonomy: Human-driven (agent-assisted)


Created with PostHog Desktop

🤖 Generated with Claude Code

…shots

Implements the "Snapshot unresolved flag reporting" and "Warning for flags local evaluation never resolves" requirements of the evaluate-flags spec (PostHog/sdk-specs#85).

- Add `FeatureFlagEvaluations.UnresolvedFlags`, with an `UnresolvedFlagReason` for each flag and the spec's fixed precedence.
- Report `local_evaluation_inconclusive` instead of `flag_missing` when a snapshot read hits an unresolved flag.
- Do not stop a condition at the first inconclusive filter, so a filter that does not match resolves the condition locally.
- Check `active` before experience continuity.
- Log one warning per flag that local evaluation can never resolve, repeated only when its definition changes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Generated-By: PostHog Desktop
Task-Id: 39793b31-809e-4689-a5d8-b4359cbb22c1
@gustavohstrassburger gustavohstrassburger self-assigned this Oct 27, 2026

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 3c0989a: 3 Should fix.

This is the only automatic review while this pull request is a draft. Comment @posthog review to review new changes sooner. Otherwise, a review of the new changes runs when you mark it ready for review.

See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.

Comment thread src/PostHog/Library/RegexHelpers.cs Resolved
Comment thread src/PostHog/Features/LocalEvaluator.cs Resolved
Comment thread src/PostHog/Features/LocalEvaluator.cs Resolved
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

posthog-dotnet Compliance Report

Date: 2026-10-28T19:59:03.027696+00:00
Duration: 124951ms

✅ All Tests Passed!

47/47 tests passed


Capture Tests

✅ 30/30 tests passed

View Details
Test Status Duration
Format Validation.Event Has Required Fields ✅ 253ms
Format Validation.Event Has Uuid ✅ 128ms
Format Validation.Event Has Lib Properties ✅ 110ms
Format Validation.Distinct Id Is String ✅ 115ms
Format Validation.Token Is Present ✅ 110ms
Format Validation.Custom Properties Preserved ✅ 118ms
Format Validation.Event Has Timestamp ✅ 111ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc ✅ 120ms
Retry Behavior.Retries On 503 ✅ 8123ms
Retry Behavior.Does Not Retry On 400 ✅ 2118ms
Retry Behavior.Does Not Retry On 401 ✅ 2114ms
Retry Behavior.Respects Retry After Header ✅ 8120ms
Retry Behavior.Implements Backoff ✅ 22126ms
Retry Behavior.Retries On 500 ✅ 6118ms
Retry Behavior.Retries On 502 ✅ 6119ms
Retry Behavior.Retries On 504 ✅ 6120ms
Retry Behavior.Max Retries Respected ✅ 22133ms
Deduplication.Generates Unique Uuids ✅ 116ms
Deduplication.Preserves Uuid On Retry ✅ 6117ms
Deduplication.Preserves Uuid And Timestamp On Retry ✅ 13131ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ✅ 6122ms
Deduplication.No Duplicate Events In Batch ✅ 117ms
Deduplication.Different Events Have Different Uuids ✅ 111ms
Compression.Sends Gzip When Enabled ✅ 110ms
Batch Format.Uses Proper Batch Structure ✅ 109ms
Batch Format.Flush With No Events Sends Nothing ✅ 106ms
Batch Format.Multiple Events Batched Together ✅ 115ms
Error Handling.Does Not Retry On 403 ✅ 2111ms
Error Handling.Does Not Retry On 413 ✅ 2112ms
Error Handling.Retries On 408 ✅ 6118ms

Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 147ms
Request Payload.Flags Request Uses V2 Query Param ✅ 110ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 109ms
Request Payload.Flags Request Omits Authorization Header ✅ 109ms
Request Payload.Token In Flags Body Matches Init ✅ 110ms
Request Payload.Groups Round Trip ✅ 114ms
Request Payload.Groups Default To Empty Object ✅ 110ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 110ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 109ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 110ms
Request Lifecycle.No Flags Request On Init Alone ✅ 4ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 109ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 214ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 114ms
Retry Behavior.Retries Flags On 502 ✅ 1115ms
Retry Behavior.Retries Flags On 504 ✅ 1112ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 212ms

@gustavohstrassburger
gustavohstrassburger marked this pull request as ready for review October 28, 2026 16:18
@gustavohstrassburger
gustavohstrassburger requested a review from a team as a code owner October 28, 2026 16:18

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed a2c64d2: 1 Should fix.

Comment thread src/PostHog/Features/LocalEvaluator.cs Resolved

@dustinbyrne dustinbyrne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cohort-regex precedence fix looks correct and the focused regression checks pass. I reproduced a separate dependency-evaluation regression where the SDK selects a later multivariate variant instead of the server’s earlier matching variant. Consider preserving outcome-dependent server uncertainty before approval.

AI-assisted review.

Comment thread src/PostHog/Features/LocalEvaluator.cs Outdated
}
catch (RequiresServerEvaluationException e)
{
throw new InconclusiveMatchException(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

F1 — [P1] Preserve uncertainty when a server-required dependency can change the variant

IMPORTANT · REQUIRED · Introduced by the full PR; unchanged by this follow-up

Location: src/PostHog/Features/LocalEvaluator.cs:1100–1105, particularly line 1102, RIGHT.

When a flag’s first condition depends on unavailable static-cohort membership, head can choose a later, different variant locally. For a person who matches the first condition on the server, the SDK now returns b where base correctly fell back and returned a.

The minimal valid definition setup is:

{
  "flags": [
    {
      "id": 1, "key": "dep", "active": true,
      "filters": {
        "groups": [{
          "properties": [{"key": "id", "type": "cohort", "value": 999}],
          "rollout_percentage": 100
        }]
      }
    },
    {
      "id": 2, "key": "target", "active": true,
      "filters": {
        "groups": [
          {
            "properties": [{
              "key": "dep", "type": "flag", "value": true,
              "operator": "flag_evaluates_to",
              "dependency_chain": ["dep"]
            }],
            "rollout_percentage": 100, "variant": "a"
          },
          {
            "properties": [],
            "rollout_percentage": 100, "variant": "b"
          }
        ],
        "multivariate": {
          "variants": [
            {"key": "a", "rollout_percentage": 50},
            {"key": "b", "rollout_percentage": 50}
          ]
        }
      }
    }
  ],
  "cohorts": {}
}

For a person belonging to static cohort 999, remote evaluation returns:

{"featureFlags":{"dep":true,"target":"a"}}

Ordinary public caller:

var snapshot = await client.EvaluateFlagsAsync(
    "member",
    new AllFeatureFlagsOptions {
        FlagKeysToEvaluate = new[] { "target" }
    },
    CancellationToken.None);

var variant = snapshot.GetFlag("target")?.VariantKey;

Fresh executed results:

Configuration Merge base Previous head Current head
Scoped target, fallback enabled a, one remote request b, none b, none
Unscoped, fallback enabled a, one request b, one request b, despite remote a
Scoped, local-only Target absent Local b Local b
Dependency locally resolves true — control a, no request Same Same
Retained GetFeatureFlagAsync a, one request b, none b, none

The new catch converts RequiresServerEvaluationException into ordinary InconclusiveMatchException. The latter permits later conditions; the former previously forced fallback. In the unscoped case, the existing local-wins merge subsequently retains unsafe local b over correct remote a.

Actual SDK exposure batches also changed from response a, remotely evaluated, to response b, locally evaluated. Applications can branch on that wrong variant. Both variants are enabled: this reproduction does not demonstrate changed boolean enablement or a security boundary.

Contract qualification: The PR intentionally discloses continuation, current Node behaves similarly, and the canonical local-evaluator contract generally permits continuing inconclusive groups. Continuation itself is not the defect.

The narrower issue is accepting b while an earlier condition can select a. The same contract requires ordered variant selection. The backend compares uncertain answers including variants, while Python preserves the server-required signal. There is no universal peer consensus, but the base/head comparison demonstrates a newly unsafe public path. Disclosure is not explicit acceptance of this particular consequence.

Regression evidence: The identical 28-case harness exercised actual SDK public methods on all three revisions with the fixture above. All compilations/runs and positive controls exited 0. python3 assertions.py dependency first confirms base returns a with one request, then fails on current head:

AssertionError:
('dependency-scoped-True-localOnly-False-localDep-False', 'b', 0)

The unscoped, local-only, legacy-getter and positive-control outcomes are independently recorded in the retained JSON results. Execution used exact SDK source with .NET 9 framework substitution; native shipping-framework xUnit execution was not performed.

Recommendation: Consider retaining the server-forcing signal while carrying unresolved_dependency, or permitting continuation only when earlier uncertainty cannot change the selected answer. A conservative repair does not require a general possible-value engine.

Add native regressions for the scoped/unscoped conflicting variants, local-only unresolved result, locally resolvable dependency and retained getter. No corrective implementation was applied or tested.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4b7c331. A dependency that needs server data now raises the server-required error again (reason unresolved_dependency), so the flag falls back to /flags as on main, and a higher-precedence reason elsewhere in the flag still wins. Regression tests: FlagWhoseDependencyNeedsServerDataFallsBackToTheServer (GetFeatureFlagAsync, scoped and unscoped EvaluateFlagsAsync all return the server's a), LocalOnlyEvaluationReportsFlagWhoseDependencyNeedsServerDataAsUnresolvedDependency, the control FlagWhoseDependencyResolvesLocallyDoesNotCallTheServer, DependencyThatRequiresServerForcesServerFallbackAsUnresolvedDependency, and new precedence cases in ReportsTheHighestPrecedenceReasonWhateverTheOrder.

@dustinbyrne
dustinbyrne requested a review from a team October 28, 2026 19:26

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 4b7c331: 1 Should fix. This pull request is large, so the review ran in 4 parts with less depth than usual.

This is review round 3 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.

Comment thread src/PostHog/Features/LocalEvaluator.cs Resolved

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed ed80bab: no findings. This pull request is large, so the review ran in 4 parts with less depth than usual.

@dustinbyrne dustinbyrne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dependency fallback and malformed-definition fixes address the earlier findings. Verified scoped/unscoped snapshots, the legacy getter and local-only behavior; current CI passes.

AI-assisted review.

@github-project-automation github-project-automation Bot moved this from In Review to Approved in Feature Flags Oct 29, 2026
@gustavohstrassburger
gustavohstrassburger merged commit 88a8941 into main Oct 29, 2026
21 checks passed
@gustavohstrassburger
gustavohstrassburger deleted the posthog/unresolved-local-flags branch October 29, 2026 17:04
@github-project-automation github-project-automation Bot moved this from Approved to Done in Feature Flags Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants