We noticed that incoming measurements slowed and clients were seeing 400 errors when submitting reports.
I didn't see any issues on the ECS services, but backend-fsn nginx logs showed that it was unable to reach the api endpoint.
However, requests made from backend-fsn had no issues (using curl).
I reloaded nginx and resolved the issue. What I don't know is why this occurred.
Jul 06 08:29:46 backend-fsn.ooni.org nginx[1046013]: backend-fsn.ooni.org ooniapi: xxx 0.0.0.0 - [06/Jul/2026:08:29:46 +0000] "POST /report HTTP/2.0" 502 snt:157 rt:0.000 uprt:0.000 "-" "miniooni/3.24.0 ooniprobe-engine/3.24.0"
Jul 06 08:29:46 backend-fsn.ooni.org nginx[1046013]: backend-fsn.ooni.org ooniapi: xxx 0.0.0.0 MISS [06/Jul/2026:08:29:46 +0000] "GET /api/v1/test-helpers HTTP/2.0" 200 snt:249 rt:0.002 uprt:0.004 "-" "ooniprobe-android-unattended/6.0.2 ooniprobe-engine/3.29.0"
Jul 06 08:29:46 backend-fsn.ooni.org nginx[1046013]: backend-fsn.ooni.org ooniapi: 2026/07/06 08:29:46 [error] 1046013#1046013: *152004051 no live upstreams while connecting to upstream, client: xxx, server: _, request: "POST /report HTTP/2.0", upstream: "https://ooniprobe.prod.ooni.io/report", host: "api.ooni.io"```
We noticed that incoming measurements slowed and clients were seeing 400 errors when submitting reports.
I didn't see any issues on the ECS services, but backend-fsn nginx logs showed that it was unable to reach the api endpoint.
However, requests made from backend-fsn had no issues (using curl).
I reloaded nginx and resolved the issue. What I don't know is why this occurred.