From 0d5a90980621b28cb468ccaedf5e2e907d78d449 Mon Sep 17 00:00:00 2001 From: Harry Cordewener Date: Mon, 28 Sep 2026 18:14:12 -0500 Subject: [PATCH] Give Traefik a GC target it can live under, and more room On 2026-09-28 a facet flood through Cloudflare took the site down while web answered /health in 1.5 ms. Traefik sat at 511.9/512 MiB with GOMEMLIMIT at 96MiB, so the Go collector ran continuously; at 62% CPU its own health probe to web missed the 3s timeout, the only backend was marked down, and every path served 503 and then 522. - GOMEMLIMIT 96MiB -> 640MiB, mem_limit 512m -> 768m: a soft target far below the working set turns memory pressure into CPU starvation. - websecure readTimeout 15s, idleTimeout 30s (defaults 60s/180s): the flood held ~2,600 established and ~1,600 CLOSE_WAIT connections. - Access log omits 429s: they were 94% of lines, so 100 MiB of rotation covered ten minutes and the start of the incident was already gone. Co-Authored-By: Claude Opus 5.5 --- deploy/compose.production.yaml | 40 +++++++++++++++++++++++++++++----- docs/deploy.md | 15 ++++++++----- 2 files changed, 44 insertions(+), 11 deletions(-) diff --git a/deploy/compose.production.yaml b/deploy/compose.production.yaml index 7092744..bcf453c 100644 --- a/deploy/compose.production.yaml +++ b/deploy/compose.production.yaml @@ -236,10 +236,16 @@ services: traefik: image: traefik:v3.7 environment: - # Keep Go collection below the container kill boundary, with transient headroom. - # The September 2026 facet flood also exhausted 256 MiB; retain the soft GC target - # and allow 512 MiB alongside the bounded filtered-listing middleware. - GOMEMLIMIT: 96MiB + # A soft target the collector aims under, set close to `mem_limit` below, never far beneath + # it. It was 96MiB against a 512 MiB limit, and on 2026-09-28 that gap was the outage: a facet + # flood through Cloudflare held ~2,600 connections open, the live heap sat far above 96MiB, and + # a Go runtime over its soft limit collects continuously (capped at about half the CPU). Traefik + # spent 62% CPU collecting at 511.9/512 MiB, its own health probe to `web` stopped completing + # inside 3s, it marked the only backend down, and every path answered 503 and then 522 while + # `web` itself served /health in 1.5 ms. A soft limit below the working set does not bound + # memory; it converts memory pressure into CPU starvation. ~83% of the limit leaves the + # collector room to act before the cgroup kills the process, which is what it is for. + GOMEMLIMIT: 640MiB # Ordering only, not `condition: service_healthy`: Compose's health-gated `depends_on` is not # well-specified against a service running `deploy.replicas` > 1, and the real gate on whether a # given `web` container receives traffic is the per-container active healthcheck on `mui-web` @@ -270,6 +276,15 @@ services: - "--entrypoints.web.http.redirections.entrypoint.permanent=true" - "--entrypoints.websecure.address=:443" + # Connections cost memory whether or not a request on them was ever served, and during the + # 2026-09-28 flood Traefik held ~2,600 established and ~1,600 in CLOSE_WAIT. The defaults are + # 60s to read a request and 180s to keep an idle keep-alive open; Cloudflare reuses its origin + # connections and reopens one transparently, so a shorter idle window costs a handshake, not a + # request. No writeTimeout: it bounds the whole response, and an MCP `crawl_run_cycle` or a + # slow reader of a large listing legitimately runs longer than any number worth setting. + - "--entrypoints.websecure.transport.respondingTimeouts.readTimeout=15s" + - "--entrypoints.websecure.transport.respondingTimeouts.idleTimeout=30s" + # Only for the ACME account behind crawler.mu-index.com, which is DNS-only and therefore gets # an ordinary Let's Encrypt certificate over HTTP-01 — unreachable through Cloudflare's proxy, # which is why the site domain uses the file-provided Origin CA certificate instead. Named @@ -290,6 +305,12 @@ services: # there for why that is considered close enough rather than replicated exactly. - "--accesslog=true" - "--accesslog.format=json" + # Everything except 429. The log exists to say what was *served*, and a request the rate + # limiter turned away reached nothing behind it. Logging them is what made the record useless + # on 2026-09-28: 60,246 of 64,240 lines in ten minutes were 429s, so the 100 MiB the `logging:` + # block below keeps covered ten minutes, and the start of the flood had rotated out before + # anyone looked. A flood's size is in Cloudflare's analytics, which see it before this box does. + - "--accesslog.filters.statuscodes=100-428,430-599" # /ping, gated on this flag, is what the HEALTHCHECK below and `traefik healthcheck` ask. - "--ping=true" @@ -329,8 +350,15 @@ services: timeout: 3s retries: 3 start_period: 10s - mem_limit: 512m - memswap_limit: 512m + # 768m, up from 512m, which a flood filled to the last 0.1 MiB. Paired with GOMEMLIMIT above: + # raising either one alone reproduces a failure — the target without the ceiling as an OOM kill, + # the ceiling without the target as the collector spiral. + # + # The host is overcommitted by these numbers and was before this change: the limits sum to about + # 4.1 GB on a 3.73 GiB box. They are ceilings, not reservations — steady use is under 1.5 GB — and + # `web`'s 2g is the one sized against a peak rather than a working set. + mem_limit: 768m + memswap_limit: 768m restart: unless-stopped watchtower: diff --git a/docs/deploy.md b/docs/deploy.md index de07b67..577254b 100644 --- a/docs/deploy.md +++ b/docs/deploy.md @@ -134,8 +134,11 @@ probe and `curl --fail http://127.0.0.1:9102/metrics`; a changed file alone prov The September 2026 incident still had two replicas and no metrics listener despite both settings having been corrected in the deployment checkout. -The proxy's `GOMEMLIMIT=96MiB` gives Go a soft collection target below its 256 MiB cgroup limit. -It is not a hard RSS cap; verify working set and OOM/restart events under load. +The proxy's `GOMEMLIMIT=640MiB` gives Go a soft collection target just below its 768 MiB cgroup +limit. It is not a hard RSS cap, and it must not sit far below the working set either: a Go process +over its soft limit collects continuously, which is how a 96MiB target took the site down on +2026-09-28 (see the Traefik comment in `deploy/compose.production.yaml`). Verify working set and +OOM/restart events under load. `MUI_METRICS_PORT` maps `GET /metrics`, in Prometheus text format. It exists because of a specific dead end: for three days at the start of September 2026 the site's memory climbed, and every @@ -884,12 +887,14 @@ Excess requests return HTTP 429. Filtered listings can therefore be throttled du flood even for ordinary visitors. The Cloudflare client header provides fairness, not authentication; the shared rate and concurrency caps still apply if a direct-origin caller spoofs that header. This route intentionally bypasses response compression to avoid concurrent compressor allocations. -Traefik keeps `GOMEMLIMIT=96MiB` and a hard 512 MiB memory limit with no container swap. +Traefik keeps `GOMEMLIMIT=640MiB` and a hard 768 MiB memory limit with no container swap, reads a +request within 15s and closes an idle connection after 30s. The access log omits 429s, so +a flood's size is read from Cloudflare's analytics rather than from this log. After merging, pull the repository in `/opt/muindex`; Watchtower does not update these files. Validate with `docker compose config --quiet`, then apply proxy changes with `docker compose up -d --no-deps traefik`. The dynamic file is watched, so middleware-only changes need no restart. Check the homepage and `/games` return 200, filtered requests appear under -`mui-games-protection@file` in access logs, excess requests receive 429, and the proxy restart -count remains stable. To roll back, restore the previous Compose overlay and remove the dynamic +`mui-games-protection@file` in access logs, excess requests receive 429 (and are absent from those +logs), and the proxy restart count remains stable. To roll back, restore the previous Compose overlay and remove the dynamic limits file, then reapply only the proxy service.