Give Traefik a GC target it can live under, and more room

deploy/compose.production.yaml

@@ -236,10 +236,16 @@ services:
   traefik:
     image: traefik:v3.7
     environment:
-      # Keep Go collection below the container kill boundary, with transient headroom.
-      # The September 2026 facet flood also exhausted 256 MiB; retain the soft GC target
-      # and allow 512 MiB alongside the bounded filtered-listing middleware.
-      GOMEMLIMIT: 96MiB
+      # A soft target the collector aims under, set close to `mem_limit` below, never far beneath
+      # it. It was 96MiB against a 512 MiB limit, and on 2026-09-28 that gap was the outage: a facet
+      # flood through Cloudflare held ~2,600 connections open, the live heap sat far above 96MiB, and
+      # a Go runtime over its soft limit collects continuously (capped at about half the CPU). Traefik
+      # spent 62% CPU collecting at 511.9/512 MiB, its own health probe to `web` stopped completing
+      # inside 3s, it marked the only backend down, and every path answered 503 and then 522 while
+      # `web` itself served /health in 1.5 ms. A soft limit below the working set does not bound
+      # memory; it converts memory pressure into CPU starvation. ~83% of the limit leaves the
+      # collector room to act before the cgroup kills the process, which is what it is for.
+      GOMEMLIMIT: 640MiB
     # Ordering only, not `condition: service_healthy`: Compose's health-gated `depends_on` is not
     # well-specified against a service running `deploy.replicas` > 1, and the real gate on whether a
     # given `web` container receives traffic is the per-container active healthcheck on `mui-web`
@@ -270,6 +276,15 @@ services:
       - "--entrypoints.web.http.redirections.entrypoint.permanent=true"
       - "--entrypoints.websecure.address=:443"
 
+      # Connections cost memory whether or not a request on them was ever served, and during the
+      # 2026-09-28 flood Traefik held ~2,600 established and ~1,600 in CLOSE_WAIT. The defaults are
+      # 60s to read a request and 180s to keep an idle keep-alive open; Cloudflare reuses its origin
+      # connections and reopens one transparently, so a shorter idle window costs a handshake, not a
+      # request. No writeTimeout: it bounds the whole response, and an MCP `crawl_run_cycle` or a
+      # slow reader of a large listing legitimately runs longer than any number worth setting.
+      - "--entrypoints.websecure.transport.respondingTimeouts.readTimeout=15s"
+      - "--entrypoints.websecure.transport.respondingTimeouts.idleTimeout=30s"
+
       # Only for the ACME account behind crawler.mu-index.com, which is DNS-only and therefore gets
       # an ordinary Let's Encrypt certificate over HTTP-01 — unreachable through Cloudflare's proxy,
       # which is why the site domain uses the file-provided Origin CA certificate instead. Named
@@ -290,6 +305,12 @@ services:
       # there for why that is considered close enough rather than replicated exactly.
       - "--accesslog=true"
       - "--accesslog.format=json"
+      # Everything except 429. The log exists to say what was *served*, and a request the rate
+      # limiter turned away reached nothing behind it. Logging them is what made the record useless
+      # on 2026-09-28: 60,246 of 64,240 lines in ten minutes were 429s, so the 100 MiB the `logging:`
+      # block below keeps covered ten minutes, and the start of the flood had rotated out before
+      # anyone looked. A flood's size is in Cloudflare's analytics, which see it before this box does.
+      - "--accesslog.filters.statuscodes=100-428,430-599"
 
       # /ping, gated on this flag, is what the HEALTHCHECK below and `traefik healthcheck` ask.
       - "--ping=true"
@@ -329,8 +350,15 @@ services:
       timeout: 3s
       retries: 3
       start_period: 10s
-    mem_limit: 512m
-    memswap_limit: 512m
+    # 768m, up from 512m, which a flood filled to the last 0.1 MiB. Paired with GOMEMLIMIT above:
+    # raising either one alone reproduces a failure — the target without the ceiling as an OOM kill,
+    # the ceiling without the target as the collector spiral.
+    #
+    # The host is overcommitted by these numbers and was before this change: the limits sum to about
+    # 4.1 GB on a 3.73 GiB box. They are ceilings, not reservations — steady use is under 1.5 GB — and
+    # `web`'s 2g is the one sized against a peak rather than a working set.
+    mem_limit: 768m
+    memswap_limit: 768m
     restart: unless-stopped
 
   watchtower:

docs/deploy.md

@@ -134,8 +134,11 @@ probe and `curl --fail http://127.0.0.1:9102/metrics`; a changed file alone prov
 The September 2026 incident still had two replicas and no metrics listener despite both settings
 having been corrected in the deployment checkout.
 
-The proxy's `GOMEMLIMIT=96MiB` gives Go a soft collection target below its 256 MiB cgroup limit.
-It is not a hard RSS cap; verify working set and OOM/restart events under load.
+The proxy's `GOMEMLIMIT=640MiB` gives Go a soft collection target just below its 768 MiB cgroup
+limit. It is not a hard RSS cap, and it must not sit far below the working set either: a Go process
+over its soft limit collects continuously, which is how a 96MiB target took the site down on
+2026-09-28 (see the Traefik comment in `deploy/compose.production.yaml`). Verify working set and
+OOM/restart events under load.
 
 `MUI_METRICS_PORT` maps `GET /metrics`, in Prometheus text format. It exists because of a specific
 dead end: for three days at the start of September 2026 the site's memory climbed, and every
@@ -884,12 +887,14 @@ Excess requests return HTTP 429. Filtered listings can therefore be throttled du
 flood even for ordinary visitors. The Cloudflare client header provides fairness, not authentication;
 the shared rate and concurrency caps still apply if a direct-origin caller spoofs that header.
 This route intentionally bypasses response compression to avoid concurrent compressor allocations.
-Traefik keeps `GOMEMLIMIT=96MiB` and a hard 512 MiB memory limit with no container swap.
+Traefik keeps `GOMEMLIMIT=640MiB` and a hard 768 MiB memory limit with no container swap, reads a
+request within 15s and closes an idle connection after 30s. The access log omits 429s, so
+a flood's size is read from Cloudflare's analytics rather than from this log.
 
 After merging, pull the repository in `/opt/muindex`; Watchtower does not update these files.
 Validate with `docker compose config --quiet`, then apply proxy changes with
 `docker compose up -d --no-deps traefik`. The dynamic file is watched, so middleware-only changes
 need no restart. Check the homepage and `/games` return 200, filtered requests appear under
-`mui-games-protection@file` in access logs, excess requests receive 429, and the proxy restart
-count remains stable. To roll back, restore the previous Compose overlay and remove the dynamic
+`mui-games-protection@file` in access logs, excess requests receive 429 (and are absent from those
+logs), and the proxy restart count remains stable. To roll back, restore the previous Compose overlay and remove the dynamic
 limits file, then reapply only the proxy service.