[posthog]

💡 Motivation and Context

Brings posthog-php into compliance with the Local Feature Flag Evaluator contract in PostHog/sdk-specs — specifically the requirements Experiment holdouts precede release conditions and Holdout membership uses backend-compatible bucketing.

The SDK compliance matrix flagged the gap (compliance/posthog-php.md, Local Feature Flag Evaluator — 🟡 Partial):

In computeFlagLocally(), before iterating condition groups, read filters.holdout; when present, hash the bucketing identity with the holdout's salt per the spec's "Holdout membership uses backend-compatible bucketing" requirement and return the holdout outcome when the identity falls inside exclusion_percentage. Keep the percentage as a float — the spec explicitly forbids truncating fractional percentages.

Today filters.holdout is dropped on the floor during local evaluation (grep -rn holdout lib/ returned nothing), so a user the backend puts in an experiment holdout gets an ordinary variant locally. That is a silent correctness divergence between local and remote evaluation for every experiment that uses a holdout.

What changed

• FeatureFlag::matchFeatureFlagProperties() resolves filters.holdout before iterating release conditions, returning holdout-<id> when the bucketing identity is held out. Because both the single-flag/bulk path (Client::computeFlagLocally()) and the dependency path (FeatureFlag::evaluateFlagDependency()) check active and then call this method, all three evaluation paths share the inactive check and the holdout precedence: an inactive flag still resolves to false, and a dependent flag comparing against "holdout-727" matches.
• FeatureFlag::holdoutHash() implements the backend hash: SHA-1 over holdout-<bucketing_value> with no separator or salt, first 15 hex digits over 0xfffffffffffffff. Neither the flag key nor the holdout id participates, so the ordinary dot-separated flag hash is deliberately not reused.
• Membership uses the inclusive comparison hash <= percentage / 100, with exclusion_percentage clamped to 0–100 as a float (no truncation) and a clamped 100 short-circuiting without computing a hash.
• A holdout that is absent, not an object, or missing/null on id or exclusion_percentage is skipped and ordinary evaluation continues unchanged.
• The holdout uses the flag-level bucketing identity, i.e. the group key for group-aggregated flags (already resolved by the caller) — per-condition aggregation does not change membership. One exception: evaluateFlagDependency() passes the parent flag's distinct id down to the dependency, so a group-aggregated dependency with a partial holdout gets bucketed by the person instead of its group key. Rollout hashing for those dependencies already works this way on main, and this PR doesn't change that.

Backwards compatibility

Additive. Flags without a (complete) filters.holdout evaluate exactly as before, and no public signature changes — composer api:check reports the public API snapshot is unchanged. Values change only for flags that actually carry a holdout, where the current locally-evaluated value is wrong and disagrees with the server.

💚 How did you test it?

New test/FeatureFlagHoldoutTest.php covers each acceptance scenario from the spec: holdout winning over an unavailable targeting property, rollout 0 and a variant override; a synthetic variant absent from filters.multivariate.variants; inactive flag staying false; no-holdout and incomplete-holdout preserving ordinary assignment; dependency evaluation comparing against the holdout string; the spec's reference hashes (user-1 → ~0.17805599206573022 held out at 20%, user-5 → ~0.6563813925994418 not); fractional percentages not truncated; inclusive/clamped boundaries at 0, -10, 100 and 150; and the group key deciding membership for a group-aggregated flag.

./vendor/bin/phpunit --bootstrap vendor/autoload.php --configuration phpunit.xml
# 751 tests, 5411 assertions, 0 failures
./vendor/bin/phpcs --standard=phpcs.xml --extensions=php lib test   # no new findings
php scripts/check-public-api.php                                    # snapshot up to date

No manual testing against a live PostHog project.

Follow-up work

• Device-id bucketing (Device identity determines membership for device-bucketed flags) is untested here because posthog-php has no device-id bucketing concept; it would land with that feature.
• The same holdout gap is recorded for posthog-ruby and is not addressed here.
• Have evaluateFlagDependency() resolve each dependency's own bucketing identity, the way posthog-js does by re-entering computeFlagValueLocally(). That would fix rollout, variant and holdout bucketing for group-aggregated dependencies together.

📝 Checklist

• I reviewed the submitted code.
• I added tests to verify the changes.
• I updated the docs if needed.
• No breaking change or entry added to the changelog.

If releasing new changes

• Ran pnpm change to generate a change intent file (added .changeset/local-flag-holdouts.md by hand, matching the existing format)

🤖 Agent context

Autonomy: Fully autonomous

Opened by a scheduled PostHog agent run that reads the SDK compliance matrices in PostHog/sdk-specs and implements one backwards-compatible gap per run. Candidate selection skipped gaps with a Breaking verdict, gaps on deprecated methods, and gaps already covered by an open PR on this repo. Holdout support was chosen over the other eligible PHP candidates (an additive getAllFlagsAndPayloads(), payload-decode warning logging, and the flag-request backoff constant) because it is a user-visible correctness divergence from the server rather than an ergonomics or diagnostics gap.

Implementation note: the holdout check was placed in matchFeatureFlagProperties() rather than Client::computeFlagLocally() as the matrix suggested, so the dependency-evaluation path gets the same semantics without duplicating the logic.

---

Created with PostHog Desktop

🤖 Generated with Claude Code

[github-actions]

posthog-php-fork_curl Compliance Report

Date: 2026-10-16T18:23:20.624577+00:00
Duration: 112457ms

⚠️ Some Tests Failed

36/47 tests passed, 11 failed

---

Capture Tests

⚠️ 19/30 tests passed, 11 failed

View Details

Test	Status	Duration
Format Validation.Event Has Required Fields	✅	37ms
Format Validation.Event Has Uuid	✅	530ms
Format Validation.Event Has Lib Properties	✅	531ms
Format Validation.Distinct Id Is String	✅	533ms
Format Validation.Token Is Present	✅	532ms
Format Validation.Custom Properties Preserved	✅	531ms
Format Validation.Event Has Timestamp	✅	532ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc	✅	533ms
Retry Behavior.Retries On 503	❌	5535ms
Retry Behavior.Does Not Retry On 400	✅	2534ms
Retry Behavior.Does Not Retry On 401	✅	2534ms
Retry Behavior.Respects Retry After Header	❌	5535ms
Retry Behavior.Implements Backoff	❌	15550ms
Retry Behavior.Retries On 500	❌	5539ms
Retry Behavior.Retries On 502	❌	5541ms
Retry Behavior.Retries On 504	❌	5534ms
Retry Behavior.Max Retries Respected	❌	15538ms
Deduplication.Generates Unique Uuids	✅	49ms
Deduplication.Preserves Uuid On Retry	❌	5539ms
Deduplication.Preserves Uuid And Timestamp On Retry	❌	10543ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry	❌	5542ms
Deduplication.No Duplicate Events In Batch	✅	536ms
Deduplication.Different Events Have Different Uuids	✅	532ms
Compression.Sends Gzip When Enabled	✅	537ms
Batch Format.Uses Proper Batch Structure	✅	527ms
Batch Format.Flush With No Events Sends Nothing	✅	519ms
Batch Format.Multiple Events Batched Together	✅	521ms
Error Handling.Does Not Retry On 403	✅	2534ms
Error Handling.Does Not Retry On 413	✅	2533ms
Error Handling.Retries On 408	❌	5536ms

Failures

retry_behavior.retries_on_503

Expected at least 3 requests, got 1

retry_behavior.respects_retry_after_header

Expected at least 2 requests, got 1

retry_behavior.implements_backoff

Expected at least 3 requests, got 1

retry_behavior.retries_on_500

Expected at least 2 requests, got 1

retry_behavior.retries_on_502

Expected at least 2 requests, got 1

retry_behavior.retries_on_504

Expected at least 2 requests, got 1

retry_behavior.max_retries_respected

Expected 4 requests, got 1

deduplication.preserves_uuid_on_retry

Need at least 2 requests to check retry

deduplication.preserves_uuid_and_timestamp_on_retry

Expected at least 3 requests, got 1

deduplication.preserves_uuid_and_timestamp_on_batch_retry

Expected at least 2 requests, got 1

error_handling.retries_on_408

Expected at least 2 requests, got 1

Feature_Flags Tests

✅ 17/17 tests passed

View Details

Test	Status	Duration
Request Payload.Request With Person Properties Device Id	✅	525ms
Request Payload.Flags Request Uses V2 Query Param	✅	522ms
Request Payload.Flags Request Hits Flags Path Not Decide	✅	521ms
Request Payload.Flags Request Omits Authorization Header	✅	523ms
Request Payload.Token In Flags Body Matches Init	✅	522ms
Request Payload.Groups Round Trip	✅	522ms
Request Payload.Groups Default To Empty Object	✅	521ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False	✅	523ms
Request Payload.Disable Geoip Omitted Defaults To False	✅	521ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key	✅	523ms
Request Lifecycle.No Flags Request On Init Alone	✅	518ms
Request Lifecycle.No Flags Request On Normal Capture	✅	518ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests	✅	525ms
Request Lifecycle.Mock Response Value Is Returned To Caller	✅	522ms
Retry Behavior.Retries Flags On 502	✅	626ms
Retry Behavior.Retries Flags On 504	✅	624ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event	✅	535ms

[github-actions]

posthog-php-lib_curl Compliance Report

Date: 2026-10-16T18:23:37.021599+00:00
Duration: 118827ms

✅ All Tests Passed!

47/47 tests passed

---

Capture Tests

✅ 30/30 tests passed

View Details

Test	Status	Duration
Format Validation.Event Has Required Fields	✅	28ms
Format Validation.Event Has Uuid	✅	521ms
Format Validation.Event Has Lib Properties	✅	524ms
Format Validation.Distinct Id Is String	✅	523ms
Format Validation.Token Is Present	✅	523ms
Format Validation.Custom Properties Preserved	✅	523ms
Format Validation.Event Has Timestamp	✅	524ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc	✅	524ms
Retry Behavior.Retries On 503	✅	5832ms
Retry Behavior.Does Not Retry On 400	✅	2525ms
Retry Behavior.Does Not Retry On 401	✅	2525ms
Retry Behavior.Respects Retry After Header	✅	8532ms
Retry Behavior.Implements Backoff	✅	16248ms
Retry Behavior.Retries On 500	✅	5632ms
Retry Behavior.Retries On 502	✅	5632ms
Retry Behavior.Retries On 504	✅	5631ms
Retry Behavior.Max Retries Respected	✅	17047ms
Deduplication.Generates Unique Uuids	✅	244ms
Deduplication.Preserves Uuid On Retry	✅	5632ms
Deduplication.Preserves Uuid And Timestamp On Retry	✅	10839ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry	✅	5635ms
Deduplication.No Duplicate Events In Batch	✅	531ms
Deduplication.Different Events Have Different Uuids	✅	525ms
Compression.Sends Gzip When Enabled	✅	524ms
Batch Format.Uses Proper Batch Structure	✅	523ms
Batch Format.Flush With No Events Sends Nothing	✅	519ms
Batch Format.Multiple Events Batched Together	✅	512ms
Error Handling.Does Not Retry On 403	✅	2527ms
Error Handling.Does Not Retry On 413	✅	2525ms
Error Handling.Retries On 408	✅	5632ms

Feature_Flags Tests

✅ 17/17 tests passed

View Details

Test	Status	Duration
Request Payload.Request With Person Properties Device Id	✅	522ms
Request Payload.Flags Request Uses V2 Query Param	✅	521ms
Request Payload.Flags Request Hits Flags Path Not Decide	✅	521ms
Request Payload.Flags Request Omits Authorization Header	✅	521ms
Request Payload.Token In Flags Body Matches Init	✅	521ms
Request Payload.Groups Round Trip	✅	521ms
Request Payload.Groups Default To Empty Object	✅	521ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False	✅	521ms
Request Payload.Disable Geoip Omitted Defaults To False	✅	521ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key	✅	521ms
Request Lifecycle.No Flags Request On Init Alone	✅	516ms
Request Lifecycle.No Flags Request On Normal Capture	✅	509ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests	✅	525ms
Request Lifecycle.Mock Response Value Is Returned To Caller	✅	522ms
Retry Behavior.Retries Flags On 502	✅	625ms
Retry Behavior.Retries Flags On 504	✅	623ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event	✅	524ms

[github-actions]

posthog-php-socket Compliance Report

Date: 2026-10-16T18:23:59.820321+00:00
Duration: 149195ms

⚠️ Some Tests Failed

36/47 tests passed, 11 failed

---

Capture Tests

⚠️ 19/30 tests passed, 11 failed

View Details

Test	Status	Duration
Format Validation.Event Has Required Fields	✅	26ms
Format Validation.Event Has Uuid	✅	521ms
Format Validation.Event Has Lib Properties	✅	523ms
Format Validation.Distinct Id Is String	✅	522ms
Format Validation.Token Is Present	✅	523ms
Format Validation.Custom Properties Preserved	✅	522ms
Format Validation.Event Has Timestamp	✅	524ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc	✅	522ms
Retry Behavior.Retries On 503	❌	9230ms
Retry Behavior.Does Not Retry On 400	✅	2528ms
Retry Behavior.Does Not Retry On 401	✅	2525ms
Retry Behavior.Respects Retry After Header	❌	9232ms
Retry Behavior.Implements Backoff	❌	19244ms
Retry Behavior.Retries On 500	❌	9231ms
Retry Behavior.Retries On 502	❌	8740ms
Retry Behavior.Retries On 504	❌	9235ms
Retry Behavior.Max Retries Respected	❌	19245ms
Deduplication.Generates Unique Uuids	✅	532ms
Deduplication.Preserves Uuid On Retry	❌	9232ms
Deduplication.Preserves Uuid And Timestamp On Retry	❌	14240ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry	❌	9238ms
Deduplication.No Duplicate Events In Batch	✅	530ms
Deduplication.Different Events Have Different Uuids	✅	524ms
Compression.Sends Gzip When Enabled	✅	522ms
Batch Format.Uses Proper Batch Structure	✅	522ms
Batch Format.Flush With No Events Sends Nothing	✅	519ms
Batch Format.Multiple Events Batched Together	✅	511ms
Error Handling.Does Not Retry On 403	✅	2524ms
Error Handling.Does Not Retry On 413	✅	2525ms
Error Handling.Retries On 408	❌	5529ms

Failures

retry_behavior.retries_on_503

Expected at least 3 requests, got 1

retry_behavior.respects_retry_after_header

Expected at least 2 requests, got 1

retry_behavior.implements_backoff

Expected at least 3 requests, got 1

retry_behavior.retries_on_500

Expected at least 2 requests, got 1

retry_behavior.retries_on_502

Expected at least 2 requests, got 1

retry_behavior.retries_on_504

Expected at least 2 requests, got 1

retry_behavior.max_retries_respected

Expected 4 requests, got 1

deduplication.preserves_uuid_on_retry

Need at least 2 requests to check retry

deduplication.preserves_uuid_and_timestamp_on_retry

Expected at least 3 requests, got 1

deduplication.preserves_uuid_and_timestamp_on_batch_retry

Expected at least 2 requests, got 1

error_handling.retries_on_408

Expected at least 2 requests, got 1

Feature_Flags Tests

✅ 17/17 tests passed

View Details

Test	Status	Duration
Request Payload.Request With Person Properties Device Id	✅	522ms
Request Payload.Flags Request Uses V2 Query Param	✅	520ms
Request Payload.Flags Request Hits Flags Path Not Decide	✅	522ms
Request Payload.Flags Request Omits Authorization Header	✅	521ms
Request Payload.Token In Flags Body Matches Init	✅	520ms
Request Payload.Groups Round Trip	✅	520ms
Request Payload.Groups Default To Empty Object	✅	520ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False	✅	520ms
Request Payload.Disable Geoip Omitted Defaults To False	✅	521ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key	✅	520ms
Request Lifecycle.No Flags Request On Init Alone	✅	516ms
Request Lifecycle.No Flags Request On Normal Capture	✅	508ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests	✅	525ms
Request Lifecycle.Mock Response Value Is Returned To Caller	✅	521ms
Retry Behavior.Retries Flags On 502	✅	623ms
Retry Behavior.Retries Flags On 504	✅	622ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event	✅	524ms

[ioannisj]

🤖 An AI agent wrote this on behalf of @ioannisj.

Fixed:

• Scoped the description's dependency claim. evaluateFlagDependency() passes the parent flag's distinct id into the dependency, so a group-aggregated dependency with a partial holdout gets bucketed by the person instead of its group key. In one evaluateFlags('user-5', ['company' => 'user-1']) call, a group flag resolves to holdout-727 while a person flag gated on it sees false. Rollout hashing already does this on main, so I left the code alone and added a follow-up to resolve each dependency's own bucketing identity like posthog-js does.

The holdoutHash() docblock says the range is [0, 1) when it can hit exactly 1.0. Comment only, so I left it.

[dustinbyrne]

The direct holdout calculation matches the specification. Consider correcting the dependency identity mismatch and adding the public-entry regression test described inline.

AI-assisted review.

[dustinbyrne]

[P1] Resolve a dependency's own identity before applying its holdout

Location: lib/FeatureFlag.php:963 (RIGHT).

The new holdout check treats $distinctId as the evaluated flag's bucketing identity, but recursive evaluateFlagDependency() passes the parent condition's identity without resolving the referenced flag's aggregation (lib/FeatureFlag.php:1224–1232). Direct group evaluation correctly supplies the group key (lib/Client.php:1411–1425).

A valid public call can therefore return contradictory results:

$flags = $client->evaluateFlags(
    'user-1',
    ['company' => 'user-5'],
    [],
    ['company' => []],
    true
);

With a group-aggregated checkout flag carrying holdout 727 at 20%, otherwise selecting control, and a person checkout-banner flag requiring checkout == control:

Result	Base, inferred	Reviewed head, inferred	Expected head
checkout	control	control	control
checkout-banner	true	false	true

The company user-5 hashes to approximately 0.6563813925994418, outside the holdout. The person user-1 hashes to approximately 0.17805599206573022, inside it. Only recursive evaluation returns holdout-727, incorrectly failing the banner condition. Both results are conclusive, so enabling normal remote fallback does not repair this.

Inherited-identity recursion existed at base, but 100% ordinary rollout and 100% control neutralize its old rollout/variant-bucketing defect in this example. Base correctly returns control through both paths; the new holdout branch alone changes the dependent flag. Impact is incorrect application gating and emitted flag values for partial-holdout dependencies whose own identity differs from the parent's; direct and remote-only evaluation remain unaffected.

The governing pinned specification requires shared dependency semantics and the flag-level group identity. The backend resolves the referenced flag's aggregation, and Node re-enters full evaluation for dependencies. Backend reference validation permits this person-parent/group-dependency configuration.

Consider resolving each dependency's own flag-level identity while retaining the original person/group context through recursion. When required context is unavailable, use the existing inconclusive path rather than substituting an unrelated identity. No new user-facing API is necessary.

Test sketch — NOT EXECUTED: Add this method to FeatureFlagHoldoutTest, using its existing holdoutFlag() builder. Explicit condition aggregation mirrors backend normalization; the noop capture consumer isolates flag-request assertions.

public function testGroupHoldoutDependencyUsesItsOwnIdentity(): void
{
    $checkout = self::holdoutFlag([
        'id' => 727,
        'exclusion_percentage' => 20,
    ]);
    $checkout['id'] = 1;
    $checkout['filters']['aggregation_group_type_index'] = 0;
    $checkout['filters']['groups'][0]['aggregation_group_type_index'] = 0;

    $banner = [
        'id' => 2,
        'key' => 'checkout-banner',
        'active' => true,
        'filters' => [
            'groups' => [[
                'aggregation_group_type_index' => null,
                'rollout_percentage' => 100,
                'properties' => [[
                    'key' => 'checkout',
                    'type' => 'flag',
                    'operator' => 'flag_evaluates_to',
                    'value' => 'control',
                    'dependency_chain' => ['checkout'],
                ]],
            ]],
        ],
    ];

    $http = new MockedHttpClient(
        'app.posthog.com',
        flagEndpointResponse: [
            'flags' => [$checkout, $banner],
            'group_type_mapping' => ['0' => 'company'],
        ]
    );
    $client = new Client(
        'test-project-key',
        ['consumer' => 'noop'],
        $http,
        secretKey: 'test-secret-key'
    );

    foreach ([true, false] as $onlyEvaluateLocally) {
        $http->calls = [];
        $flags = $client->evaluateFlags(
            'user-1',
            ['company' => 'user-5'],
            [],
            ['company' => []],
            $onlyEvaluateLocally
        );

        self::assertSame('control', $flags->getFlag('checkout'));
        self::assertTrue($flags->getFlag('checkout-banner'));
        self::assertSame([], $http->calls);
    }
}

Expected: both values match the assertions without remote evaluation in either mode. Reviewed-source implication: checkout is control, but banner is false, failing the second assertion. The equivalent fixture and builder are expected to pass against base and after correction. These outcomes are inferred, not executed.

Proposed command, not run:

./vendor/bin/phpunit --bootstrap vendor/autoload.php --configuration phpunit.xml --filter testGroupHoldoutDependencyUsesItsOwnIdentity

Also cover the opposite membership direction: person user-5, company user-1, dependency expecting holdout-727, with checkout holdout-727 and banner true. This companion tests newly supported holdouts; unlike the control fixture, it is not a base-passing regression.

[ioannisj]

🤖 An AI agent wrote this on behalf of @ioannisj.

Fixed in 2e8dc97. evaluateFlagDependency() now carries the root flag's identity aggregation through the recursion, and a dependency with an active holdout aggregated differently from that identity goes inconclusive instead of being bucketed by the parent's distinct id.

So evaluateFlags('user-1', ['company' => 'user-5']) no longer returns holdout-727 for a group-aggregated checkout, and checkout-banner falls back to /flags when remote evaluation is allowed.

I went with deferring to the server rather than resolving the dependency's own group key locally, since that key isn't reachable inside the recursion today. Covered both membership directions plus the same-aggregation and nested-dependency cases in test/FeatureFlagHoldoutTest.php. No public API change, the identity parameter lives on private workers and api/public-api.json is untouched.

[github-actions]

This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in another week.

[marandaneto]

Automated advisory code review.

[marandaneto]

blocking: Preserve aggregation context through cohort dependencies — matchCohort() → matchPropertyGroup() still calls this public wrapper, so passing false disables the new aggregation guard on that path. With a 20% company-aggregated holdout, person user-1 and company user-5, checkout correctly resolves to control, but a person flag targeting a cohort that requires checkout = holdout-727 incorrectly evaluates to true: dependency evaluation hashes the person ID instead of the company key. Please carry identity aggregation through cohort/property-group recursion too, so this path also defers when the identities differ. Reproduction: reproduced — ./vendor/bin/phpunit --bootstrap vendor/autoload.php --configuration phpunit.xml --no-coverage --do-not-cache-result test/HoldoutCohortReviewTest.php fails on 2e8dc97 with banner=true despite checkout=control, and passes on merge base 5f16426.

[marandaneto]

approving to unblock but found #254 (comment)