Repository navigation
Glific: Fix HTTP 520 errors #1142
Description
Is your feature request related to a problem?
Glific is experiencing intermittent HTTP 520 errors (~5–6 times per day) on the POST /api/v1/llm/call endpoint. The issue likely arises from idle-timeout settings in the connection layer rather than application code, affecting overall reliability.
More details here
Describe the solution you'd like
- Update
Dockerfileto add--timeout-keep-alive 180to the uvicorn CMD. - Modify
core/rate_monitor.pyto includesocket_timeout=1andsocket_connect_timeout=1for the Redis client. - Raise ALB connection idle timeout to 120s.
- Enable access logs to S3 for future diagnostics.
Original issue
Context
Glific intermittently receives HTTP 520s (~5–6/day) on POST /api/v1/llm/call across multiple orgs. App logs show healthy sub-second processing and complete job-enqueue traces for the affected windows — the failures probably happen at the connection layer (Cloudflare ↔ ALB ↔ uvicorn), not in application code. Low traffic (~1.2 req/min) means keepalive connections sit idle constantly, triggering the classic idle-timeout race with default settings.
Ref: internal investigation doc (debug.md), Glific report from Priyanshu, 2026-08-18 failures.
Code changes
-
Dockerfile: add--timeout-keep-alive 180to the uvicorn CMD (must exceed ALB idle timeout) -
core/rate_monitor.py: addsocket_timeout=1,socket_connect_timeout=1to the Redis client — currently it can block requests indefinitely
Ops changes (done/tracking)
- ALB: raise connection idle timeout to 120s (> Cloudflare's 90s origin keepalive)
- ALB: enable access logs to S3 to diagnose the next occurrence (will cost extra - decide if should be done now or later)
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
[-]Intermittent Cloudflare 520s on Glific-facing endpoints — connection-layer hardening[/-][+]Glific: Fix HTTP 520 errors[/+]Prajna1999 commentedon Aug 28, 2026
With this 1sec socket_timeout if Redis is just slow (GC pause, network blip, connection pool exhaustion) rather than down, calls will time out and raise redis.exceptions.TimeoutError/ConnectionError . Need some more testing to check what should be the ideal number for our use case