You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Unbounded concurrent Monitoring API requests exhaust memory and OOM-kill the container #537
Root cause confirmed: each scrape fetches time series for every metric
descriptor of every configured project concurrently, with no limit — one
goroutine + HTTP request + JSON decode per descriptor, per project. When google.projects.filter (or a long google.project-ids list) resolves to
many projects, this spawns far more concurrent Monitoring API requests than
a memory-constrained pod can service at once, leading to OOM kills. This is
worse in containers specifically, since GOMAXPROCS reports the host's
vCPUs rather than the container's cgroup CPU limit — far more goroutines get
scheduled than the pod's actual quota, and CPU throttling stalls many of
them mid-decode with their buffers unfreed.
Fix: added a new flag, monitoring.max-concurrency (default 0,
unbounded — no behavior change for existing users unless set), which caps
the number of concurrent Monitoring API time series requests across all
projects in a single scrape via a shared semaphore. This bounds memory
usage regardless of how many projects are resolved.
changed the title [-][BUG] Unbounded goroutine fan-out multi-project scrapes causes memory exhaustion/ container OOM kills[/-][+]Unbounded concurrent Monitoring API requests exhaust memory and OOM-kill the container[/+]on Aug 7, 2026
Activity
ntmspavan commentedon Aug 7, 2026
Fix: added a new flag,
monitoring.max-concurrency(default0,unbounded — no behavior change for existing users unless set), which caps
the number of concurrent Monitoring API time series requests across all
projects in a single scrape via a shared semaphore. This bounds memory
usage regardless of how many projects are resolved.
#536
[-][BUG] Unbounded goroutine fan-out multi-project scrapes causes memory exhaustion/ container OOM kills[/-][+]Unbounded concurrent Monitoring API requests exhaust memory and OOM-kill the container[/+]