Skip to content

Unbounded concurrent Monitoring API requests exhaust memory and OOM-kill the container #537

Open

Description

@ntmspavan

Root cause confirmed: each scrape fetches time series for every metric
descriptor of every configured project concurrently, with no limit — one
goroutine + HTTP request + JSON decode per descriptor, per project. When
google.projects.filter (or a long google.project-ids list) resolves to
many projects, this spawns far more concurrent Monitoring API requests than
a memory-constrained pod can service at once, leading to OOM kills. This is
worse in containers specifically, since GOMAXPROCS reports the host's
vCPUs rather than the container's cgroup CPU limit — far more goroutines get
scheduled than the pod's actual quota, and CPU throttling stalls many of
them mid-decode with their buffers unfreed.

Activity

  1. ntmspavan commented on Aug 7, 2026

    @ntmspavan
    ContributorAuthor

    Fix: added a new flag, monitoring.max-concurrency (default 0,
    unbounded — no behavior change for existing users unless set), which caps
    the number of concurrent Monitoring API time series requests across all
    projects in a single scrape via a shared semaphore. This bounds memory
    usage regardless of how many projects are resolved.

    #536

  2. changed the title [-][BUG] Unbounded goroutine fan-out multi-project scrapes causes memory exhaustion/ container OOM kills[/-] [+]Unbounded concurrent Monitoring API requests exhaust memory and OOM-kill the container[/+] on Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions