Skip to content

fix: resend complete batches on socket retries - #264

Open
pauldambra wants to merge 1 commit into
mainfrom
posthog/fix-socket-retry-state
Open

pauldambra wants to merge 1 commit into
mainfrom
posthog/fix-socket-retry-state

Conversation

@pauldambra

Copy link
Copy Markdown
Member

🤖 Automated PR prepared with PostHog Desktop.

💡 Motivation and Context

Applications that use the Socket consumer fail to deliver a batch after a transient HTTP rejection.

  • Each retry now sends the complete original request, so event UUIDs and timestamps stay stable.
  • A failed replacement connection stops safely.
  • HTTP 408 and Retry-After support remain separate follow-up work.

💚 How did you test it?

Parameterized 503 and 429 cases failed before the repair and passed after it. They check complete equal bodies and an empty second flush.

The PHP suite, changed-file style check, and public API check passed. The full compliance harness was not rerun for this repair.

📝 Checklist

  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran pnpm change to generate a change intent file

Corepack could not load pnpm because certificate validation failed. A patch change intent follows the existing file format.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

PostHog Desktop with GPT-6 wrote the change. A focused GPT-6.1 Sol review accepted the repair. Skills: pr-shepherd, writing-tests, writing-pr-descriptions, and paul-pair. Human review is required.


Created with PostHog Desktop

Generated-By: PostHog Desktop
Task-Id: 5681be44-4986-48db-b140-1bda88f75f96
@pauldambra
pauldambra requested a review from a team as a code owner October 16, 2026 14:45
@pauldambra pauldambra self-assigned this Oct 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

posthog-php-fork_curl Compliance Report

Date: 2026-10-16T14:47:51.340969+00:00
Duration: 112407ms

⚠️ Some Tests Failed

36/47 tests passed, 11 failed


Capture Tests

⚠️ 19/30 tests passed, 11 failed

View Details
Test Status Duration
Format Validation.Event Has Required Fields ✅ 35ms
Format Validation.Event Has Uuid ✅ 527ms
Format Validation.Event Has Lib Properties ✅ 533ms
Format Validation.Distinct Id Is String ✅ 529ms
Format Validation.Token Is Present ✅ 532ms
Format Validation.Custom Properties Preserved ✅ 530ms
Format Validation.Event Has Timestamp ✅ 530ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc ✅ 530ms
Retry Behavior.Retries On 503 ❌ 5532ms
Retry Behavior.Does Not Retry On 400 ✅ 2534ms
Retry Behavior.Does Not Retry On 401 ✅ 2534ms
Retry Behavior.Respects Retry After Header ❌ 5539ms
Retry Behavior.Implements Backoff ❌ 15546ms
Retry Behavior.Retries On 500 ❌ 5537ms
Retry Behavior.Retries On 502 ❌ 5539ms
Retry Behavior.Retries On 504 ❌ 5536ms
Retry Behavior.Max Retries Respected ❌ 15543ms
Deduplication.Generates Unique Uuids ✅ 42ms
Deduplication.Preserves Uuid On Retry ❌ 5536ms
Deduplication.Preserves Uuid And Timestamp On Retry ❌ 10541ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ❌ 5539ms
Deduplication.No Duplicate Events In Batch ✅ 537ms
Deduplication.Different Events Have Different Uuids ✅ 532ms
Compression.Sends Gzip When Enabled ✅ 533ms
Batch Format.Uses Proper Batch Structure ✅ 529ms
Batch Format.Flush With No Events Sends Nothing ✅ 517ms
Batch Format.Multiple Events Batched Together ✅ 521ms
Error Handling.Does Not Retry On 403 ✅ 2532ms
Error Handling.Does Not Retry On 413 ✅ 2532ms
Error Handling.Retries On 408 ❌ 5534ms

Failures

retry_behavior.retries_on_503

Expected at least 3 requests, got 1

retry_behavior.respects_retry_after_header

Expected at least 2 requests, got 1

retry_behavior.implements_backoff

Expected at least 3 requests, got 1

retry_behavior.retries_on_500

Expected at least 2 requests, got 1

retry_behavior.retries_on_502

Expected at least 2 requests, got 1

retry_behavior.retries_on_504

Expected at least 2 requests, got 1

retry_behavior.max_retries_respected

Expected 4 requests, got 1

deduplication.preserves_uuid_on_retry

Need at least 2 requests to check retry

deduplication.preserves_uuid_and_timestamp_on_retry

Expected at least 3 requests, got 1

deduplication.preserves_uuid_and_timestamp_on_batch_retry

Expected at least 2 requests, got 1

error_handling.retries_on_408

Expected at least 2 requests, got 1

Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 523ms
Request Payload.Flags Request Uses V2 Query Param ✅ 521ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 522ms
Request Payload.Flags Request Omits Authorization Header ✅ 521ms
Request Payload.Token In Flags Body Matches Init ✅ 521ms
Request Payload.Groups Round Trip ✅ 523ms
Request Payload.Groups Default To Empty Object ✅ 521ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 521ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 521ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 523ms
Request Lifecycle.No Flags Request On Init Alone ✅ 516ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 518ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 525ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 523ms
Retry Behavior.Retries Flags On 502 ✅ 623ms
Retry Behavior.Retries Flags On 504 ✅ 623ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 533ms

@github-actions

Copy link
Copy Markdown
Contributor

posthog-php-lib_curl Compliance Report

Date: 2026-10-16T14:48:11.421192+00:00
Duration: 118080ms

✅ All Tests Passed!

47/47 tests passed


Capture Tests

✅ 30/30 tests passed

View Details
Test Status Duration
Format Validation.Event Has Required Fields ✅ 22ms
Format Validation.Event Has Uuid ✅ 518ms
Format Validation.Event Has Lib Properties ✅ 521ms
Format Validation.Distinct Id Is String ✅ 521ms
Format Validation.Token Is Present ✅ 521ms
Format Validation.Custom Properties Preserved ✅ 522ms
Format Validation.Event Has Timestamp ✅ 521ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc ✅ 520ms
Retry Behavior.Retries On 503 ✅ 5830ms
Retry Behavior.Does Not Retry On 400 ✅ 2522ms
Retry Behavior.Does Not Retry On 401 ✅ 2522ms
Retry Behavior.Respects Retry After Header ✅ 8528ms
Retry Behavior.Implements Backoff ✅ 16235ms
Retry Behavior.Retries On 500 ✅ 5136ms
Retry Behavior.Retries On 502 ✅ 5633ms
Retry Behavior.Retries On 504 ✅ 5631ms
Retry Behavior.Max Retries Respected ✅ 17044ms
Deduplication.Generates Unique Uuids ✅ 227ms
Deduplication.Preserves Uuid On Retry ✅ 5628ms
Deduplication.Preserves Uuid And Timestamp On Retry ✅ 10836ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ✅ 5633ms
Deduplication.No Duplicate Events In Batch ✅ 528ms
Deduplication.Different Events Have Different Uuids ✅ 521ms
Compression.Sends Gzip When Enabled ✅ 523ms
Batch Format.Uses Proper Batch Structure ✅ 521ms
Batch Format.Flush With No Events Sends Nothing ✅ 516ms
Batch Format.Multiple Events Batched Together ✅ 512ms
Error Handling.Does Not Retry On 403 ✅ 2524ms
Error Handling.Does Not Retry On 413 ✅ 2523ms
Error Handling.Retries On 408 ✅ 5630ms

Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 520ms
Request Payload.Flags Request Uses V2 Query Param ✅ 518ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 519ms
Request Payload.Flags Request Omits Authorization Header ✅ 519ms
Request Payload.Token In Flags Body Matches Init ✅ 521ms
Request Payload.Groups Round Trip ✅ 521ms
Request Payload.Groups Default To Empty Object ✅ 518ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 521ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 519ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 520ms
Request Lifecycle.No Flags Request On Init Alone ✅ 515ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 508ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 524ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 519ms
Retry Behavior.Retries Flags On 502 ✅ 622ms
Retry Behavior.Retries Flags On 504 ✅ 621ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 521ms

@github-actions

Copy link
Copy Markdown
Contributor

posthog-php-socket Compliance Report

Date: 2026-10-16T14:48:12.366983+00:00
Duration: 114901ms

⚠️ Some Tests Failed

45/47 tests passed, 2 failed


Capture Tests

⚠️ 28/30 tests passed, 2 failed

View Details
Test Status Duration
Format Validation.Event Has Required Fields ✅ 21ms
Format Validation.Event Has Uuid ✅ 516ms
Format Validation.Event Has Lib Properties ✅ 518ms
Format Validation.Distinct Id Is String ✅ 517ms
Format Validation.Token Is Present ✅ 517ms
Format Validation.Custom Properties Preserved ✅ 518ms
Format Validation.Event Has Timestamp ✅ 517ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc ✅ 517ms
Retry Behavior.Retries On 503 ✅ 5824ms
Retry Behavior.Does Not Retry On 400 ✅ 2520ms
Retry Behavior.Does Not Retry On 401 ✅ 2520ms
Retry Behavior.Respects Retry After Header ❌ 5625ms
Retry Behavior.Implements Backoff ✅ 16240ms
Retry Behavior.Retries On 500 ✅ 5626ms
Retry Behavior.Retries On 502 ✅ 5626ms
Retry Behavior.Retries On 504 ✅ 5623ms
Retry Behavior.Max Retries Respected ✅ 16241ms
Deduplication.Generates Unique Uuids ✅ 522ms
Deduplication.Preserves Uuid On Retry ✅ 5625ms
Deduplication.Preserves Uuid And Timestamp On Retry ✅ 10837ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ✅ 5624ms
Deduplication.No Duplicate Events In Batch ✅ 522ms
Deduplication.Different Events Have Different Uuids ✅ 519ms
Compression.Sends Gzip When Enabled ✅ 517ms
Batch Format.Uses Proper Batch Structure ✅ 517ms
Batch Format.Flush With No Events Sends Nothing ✅ 514ms
Batch Format.Multiple Events Batched Together ✅ 510ms
Error Handling.Does Not Retry On 403 ✅ 2519ms
Error Handling.Does Not Retry On 413 ✅ 2520ms
Error Handling.Retries On 408 ❌ 5523ms

Failures

retry_behavior.respects_retry_after_header

Retry delay too short: 102ms < 2500ms

error_handling.retries_on_408

Expected at least 2 requests, got 1

Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 518ms
Request Payload.Flags Request Uses V2 Query Param ✅ 516ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 516ms
Request Payload.Flags Request Omits Authorization Header ✅ 517ms
Request Payload.Token In Flags Body Matches Init ✅ 516ms
Request Payload.Groups Round Trip ✅ 517ms
Request Payload.Groups Default To Empty Object ✅ 517ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 518ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 517ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 517ms
Request Lifecycle.No Flags Request On Init Alone ✅ 513ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 507ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 520ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 517ms
Retry Behavior.Retries Flags On 502 ✅ 620ms
Retry Behavior.Retries Flags On 504 ✅ 620ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 519ms

@pauldambra
pauldambra added this pull request to stack #266 October 16, 2026 15:57

@marandaneto marandaneto left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated advisory code review. Verdict: needs attention.

Comment thread lib/Consumer/Socket.php
$backoff = 100; // Set initial waiting time to 100ms

while (true) {
$bytes_written = 0;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocking: Bound retries for repeated redirects — Resetting the write offset now sends a complete POST after every 3xx response, but those responses bypass both the backoff-limit check and sleep below. A host/proxy continually returning 301/302 therefore makes flush() (including destructor flushing) send requests indefinitely and block the customer process. Previously the replacement connection received no request, timed out, and reached the retry limit through status 0. Please terminate unsupported responses or enforce the retry bound for every repeated request. Reproduction: reproduced — testRedirectResponsesDoNotBypassRetryBudget observed nine requests with maximum_backoff_duration=200 on the reviewed head, despite that budget permitting only two retryable attempts; the same test passed at the exact merge base.

@marandaneto
marandaneto requested a review from a team October 16, 2026 18:18

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants