logger: retry Slack webhook rate limits

[chrismaree]

What Changed

• Added bounded retry handling for Slack webhook HTTP 429 responses.
• Honors Slack Retry-After headers, including numeric and date forms, with a 60-second safety cap.
• Retries each rate-limited post up to two times before surfacing the existing TransportError.
• Does not retry 5xx responses or connection-level failures because incoming webhook posts are not idempotent and delivery is ambiguous in those cases.
• Added focused unit coverage for retry success, retry exhaustion, ambiguous failures, and Retry-After capping.

Why

• Recent transport failures consistently showed Slack webhook HTTP 429 responses.
• Slack documents that incoming webhooks are limited to about one message per second, with short bursts allowed, and that 429 responses include the number of seconds to wait in Retry-After.
• Before this change, every 429 immediately became a TransportError. Waiting and retrying gives Slack a chance to accept the rejected post instead of surfacing a delivery failure.

Impact

• Directly addresses the observed Slack rate-limit failure mode.
• Keeps Slack routing and payload formatting unchanged.
• Keeps the final failure behavior unchanged after retry exhaustion.
• Avoids introducing duplicate Slack messages for ambiguous 5xx or connection failures.

High risk Sections to review with detail

• postWithRetry in SlackTransport.ts: retry count and delay handling determine how long a logger write can wait.
• getSlackPostRetryDelaySeconds: parses Slack response headers and caps a single wait at 60 seconds.
• Split-message Slack payload flow: each rejected chunk uses the same bounded retry helper.

Validation

• Repository pre-commit hooks ran scoped ESLint and formatting checks successfully.
• Ran git diff --check successfully.
• Did not run builds or tests, per repository instructions not to run them unless explicitly requested.

Docs

• No separate repository documentation update was needed; the behavior and safety boundary are documented in code, tests, and this PR.

[md0x]

isRetryableSlackPostError only returns true when the error carries an HTTP status in the retryable set. Requests that fail without a response — timeouts, ECONNRESET, DNS failures — have status === undefined, so they are not retried and surface as a TransportError on the first attempt. Since connection-level failures are among the most common transient Slack delivery issues (and the PR title targets transient failures broadly), consider retrying those too, or documenting that only status-coded failures are in scope.

[chrismaree]

Agreed that the original title and behavior were broader than the safe retry boundary. The incident review showed that every inspected page was an HTTP 429, so I narrowed retries to 429 only and added coverage proving connection-level failures fail fast. Slack webhook posts are not idempotent, so a no-response failure is delivery-ambiguous and retrying it could duplicate the alert. I also updated the PR title and description to state that rate limits, rather than every transient failure, are in scope.

_{Chris's Codex agent · automated}

[md0x]

Retrying 5xx responses can lead to duplicate Slack messages: a 500/502/503/504 from an intermediary can occur after Slack already accepted the payload, and the retry re-posts the same chunk. Low impact for monitoring alerts, but worth noting since webhook posts aren't idempotent (429 is safe here since it's rejected, not delivered).

[chrismaree]

Agreed. I removed retries for 500/502/503/504 and added a test asserting that a 503 is surfaced after one attempt. The helper now retries only Slack 429, where the webhook post was rejected rather than ambiguously accepted. I also raised the Retry-After safety cap from 5 to 60 seconds so the implementation can follow Slack's documented backoff signal during the rate-limit bursts seen in PagerDuty.

_{Chris's Codex agent · automated}

[md0x]

I reviewed and didn't find anything! LGTM