Cache export fails with a fatal 400 when parallel blob PUTs stall past a registry/proxy inactivity timeout on slow uplinks

### Contributing guidelines

- [x] I've read the [contributing guidelines](https://github.com/docker/buildx/blob/master/.github/CONTRIBUTING.md) and wholeheartedly agree

### I've found a bug and checked that ...

- [ ] ... the documentation does not mention anything about my problem
- [x] ... there are no open or closed issues that are related to my problem

### Description

On a bandwidth-limited or high-latency uplink, `--cache-to type=registry` intermittently fails with:

```
ERROR: failed to build: failed to solve: error writing layer blob:
unexpected status from PUT request to https://.../blobs/uploads/<uuid>?...&digest=sha256:...: 400 Bad Request
```

The failure is not a registry rejecting the content. It is a reverse proxy in front of the registry closing a blob-upload connection that BuildKit opened but left idle, and BuildKit treating the resulting `400` as fatal.


### Expected behaviour

The `docker buildx build` succeeds pushing to the server.

### Actual behaviour

The build fails with a 400 due to a timeout:

```
[2026-09-04 14:48:02]  > exporting cache to registry:
[2026-09-04 14:48:02] ------
[2026-09-04 14:48:02] ERROR: failed to build: failed to solve: error writing layer blob: unexpected status from PUT request to https://registry.gitlab.com/v2/gitlab-com/gl-security/product-security/vulnerability-management/vulnerability-management-internal/test-400-docker-timeout/buildkit-cache-repro/cache-1788504314/blobs/uploads/35856cec-2407-4559-8edb-fd65ac0ce48e?_state=FwZgrYT3V1ovNxDZxJ46rS3v_IYjGBlE-laiuFistit7Ik5hbWUiOiJnaXRsYWItY29tL2dsLXNlY3VyaXR5L3Byb2R1Y3Qtc2VjdXJpdHkvdnVsbmVyYWJpbGl0eS1tYW5hZ2VtZW50L3Z1bG5lcmFiaWxpdHktbWFuYWdlbWVudC1pbnRlcm5hbC90ZXN0LTQwMC1kb2NrZXItdGltZW91dC9idWlsZGtpdC1jYWNoZS1yZXByby9jYWNoZS0xNzg4NTA0MzE0IiwiVVVJRCI6IjM1ODU2Y2VjLTI0MDctNDU1OS04ZWRiLWZkNjVhYzBjZTQ4ZSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyNi0wOS0wNFQwNjo0NTozMy45MTkyMDk3NTRaIn0%3D&digest=sha256%3A26db0177503fd959b37402c65cc73e755a5f93441fba74e048f9bd118f27fa1b: 400 Bad Request: <!DOCTYPE html>
```

#### What happens on the wire

We captured a pcap while pushing an image to from Australia to the US (`registry.gitlab.com`) under these conditions: 

* ~260ms round-trip time (RTT) between the client and the registry, which is fronted by HAProxy with a 90s `timeout client`.
*  ~1 GB of layers total over a ~57 Mbit/s uplink

During this time, we saw:

- BuildKit opened ~8 concurrent blob PUTs. The image export and the cache export ran as two separate dispatch trees against the same host, each ~4 wide (see concurrency note below).
- About four connections actively uploaded and made continuous progress.
- The other four sent only a few KB, then went completely silent. They stayed alive at the TCP layer via keep-alives every 15s, all ACKed, but carried no HTTP body data.
- After ~90s of body inactivity the proxy returned `400` and closed each idle connection. containerd surfaced it as the fatal error above and the whole build failed.

The link and path were healthy: keep-alives answered, ~1.5% retransmission. The connections were starved, not broken. On a fast, low-RTT runner the entire export completes well under 90s and nothing idles that long, which is why this only reproduces from far, slow-uplink locations.

#### Why the connections go idle

In vendored containerd `remotes/docker/pusher.go`, a blob upload is a single monolithic streaming PUT (`// TODO: Support chunked upload`). `push()` sends the POST for the upload location, then immediately launches the PUT goroutine with an `io.Pipe` body. The request is on the wire before any bytes exist; bytes only flow once BuildKit copies content into `pushWriter.Write`. A PUT whose content copy is queued behind other uploads therefore sits open with a silent body and no heartbeat, so an upstream inactivity timeout fires.

#### Two things that turn a slow upload into a hard failure

1. **Push concurrency is not tunable.** `util/resolver/limited/group.go` hardcodes `DefaultMaxConcurrency = 4` per registry domain. `SetMaxConcurrency` exists but is never called, so a slow-link user cannot lower the fan-out without a custom build. Because image export and cache export are separate trees, the effective fan-out to one host is ~8.
2. **The proxy's `400` is non-retryable.** `remotes/docker/resolver.go` `retryRequest` retries 401/408/429/500/503/504 but not 400, so an inactivity-timeout `400` aborts the build instead of retrying the blob.

#### Reproduction

1. Run a registry (or reverse proxy) with a client inactivity timeout around 60–90s on blob-upload PUTs.
2. From a client whose upstream bandwidth makes the full cache export take longer than that timeout (throttle egress or use a distant host), build an image with several large layers and export cache:

```dockerfile
# syntax=docker/dockerfile:1.7
FROM alpine:3.20
RUN for i in $(seq 1 20); do dd if=/dev/urandom of=/layer$i.bin bs=1M count=100; done
```

```bash
docker buildx build \
  --push -t <registry>/<repo>:test \
  --cache-to type=registry,ref=<registry>/<repo>/cache,mode=max \
  --cache-from type=registry,ref=<registry>/<repo>/cache .
```

With the uplink throttled so the export exceeds the proxy timeout, some cache-blob PUTs stall and fail with `400`. Disabling `--cache-to` avoids it (fewer concurrent PUTs). `BUILDKIT_MAX_PARALLELISM` does not help, since it limits solver steps, not push fan-out.

#### Expected

BuildKit should not leave blob-upload PUTs open with an idle body while other uploads saturate the link, and a single slow blob should not fail the whole build.

#### Suggestions

- Do not begin the PUT (or send the request body) until the content copy for that blob is actually scheduled, so idle in-flight PUTs do not exist.
- Implement chunked/resumable upload (the standing TODO), so a stalled blob resumes and PATCH chunks keep the connection active.
- Expose registry push concurrency (wire `SetMaxConcurrency`) so slow-link users can set it to 1–2.
- Add a client-side idle/write deadline that aborts and retries a stalled blob rather than waiting on the server to close it.


### Buildx version

github.com/docker/buildx 0.13.1+ds1 0.13.1+ds1-3

### Docker info

```text
Client:
 Version:    26.1.5+dfsg1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.13.1+ds1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 26.1.5+dfsg1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1.7.24~ds1-6+deb13u1
 runc version: 1.1.15+ds1-2+b4
 init version:
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.12.107+deb13-cloud-amd64
 Operating System: Debian GNU/Linux 13 (trixie)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.759GiB
 Name: ubuntu-test
 ID: 59e57859-7bfc-4480-bb29-efe628bd70f1
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
```

### Builders list

```text
NAME/NODE     DRIVER/ENDPOINT   STATUS    BUILDKIT         PLATFORMS
default*      docker
 \_ default    \_ default       running   v0.0.0+unknown   linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
```

### Configuration

## Testing from Perth NBN Connection.

#### Outcome

- First attempt hit a 400.
- subsequent attempts were aborted.
- Pcap captured 400

#### Test process

1. Create large docker image
```dockerfile
# syntax=docker/dockerfile:1.7

FROM alpine:3.20
RUN apk add --no-cache coreutils

RUN dd if=/dev/urandom of=/layer01.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer02.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer03.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer04.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer05.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer06.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer07.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer08.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer09.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer10.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer01.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer02.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer03.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer04.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer05.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer06.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer07.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer08.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer09.bin bs=1M count=100 status=progress
RUN dd if=/dev/urandom of=/layer10.bin bs=1M count=100 status=progress

CMD ["sh", "-lc", "ls -lh /layer*.bin && sleep 3600"]
```
1. Setup cache destination

```bash
export CACHE=registry.gitlab.com/tmike_ultimate_group/zd722231/zd722231/buildkit-cache-repro/cache-$(date +%s)
export IMAGE=registry.gitlab.com/tmike_ultimate_group/zd722231/zd722231/buildkit-large-repro:test
```
2. Run 10 back to back image pushes with Pcaps 
```bash
#!/bin/zsh
set -euo pipefail

# All output for this run goes into its own directory
RUN_DIR="run-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$RUN_DIR"
echo "Run directory: $RUN_DIR"

# Pre-authenticate sudo NOW (synchronously) so the backgrounded tcpdump
# below never sits there silently waiting on a password prompt.
echo "Authenticating sudo for tcpdump..."
sudo -v

# Auto-detect the active network interface (macOS doesn't support -i any
# the way Linux does).
IFACE=$(route get default 2>/dev/null | awk '/interface:/{print $2}')
if [[ -z "$IFACE" ]]; then
  echo "Could not auto-detect interface; defaulting to en0"
  IFACE="en0"
fi
echo "Capturing on interface: $IFACE"

PCAP="$RUN_DIR/capture.pcap"
echo "Starting tcpdump -> $PCAP"
sudo tcpdump -i "$IFACE" -U -w "$PCAP" host registry.gitlab.com and port 443 &
TCPDUMP_PID=$!
sleep 1
if ! kill -0 "$TCPDUMP_PID" 2>/dev/null; then
  echo "WARNING: tcpdump did not start successfully"
fi
trap 'sudo kill $TCPDUMP_PID 2>/dev/null' EXIT

for i in {1..10}; do
  echo "=== Build run $i of 10 ==="
  LOGFILE="$RUN_DIR/build-${i}.log"

  docker buildx build --progress=plain --provenance=false --push -t "$IMAGE" --cache-to type=registry,ref="$CACHE",mode=max . 2>&1 \
    | perl -MPOSIX=strftime -pe '$|=1; print strftime("[%Y-%m-%d %H:%M:%S] ", localtime);' \
    | tee "$LOGFILE"

  echo "=== Finished run $i, log: $LOGFILE ==="
done

echo "All done. Everything for this run is in: $RUN_DIR"
```

3. Interogate logs for `400 Bad Request`


### Build logs

```text
[build-1.log](https://github.com/user-attachments/files/31843066/build-1.log)
```

### Additional info

_No response_

Navigation Menu

Uh oh!

Cache export fails with a fatal 400 when parallel blob PUTs stall past a registry/proxy inactivity timeout on slow uplinks #4057

Contributing guidelines

I've found a bug and checked that ...

Description

Expected behaviour

Actual behaviour

What happens on the wire

Why the connections go idle

Two things that turn a slow upload into a hard failure

Reproduction

Expected

Suggestions

Buildx version

Docker info

Builders list

Configuration

Testing from Perth NBN Connection.

Outcome

Test process

Build logs

Additional info

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Uh oh!

Cache export fails with a fatal 400 when parallel blob PUTs stall past a registry/proxy inactivity timeout on slow uplinks #4057

Description

Contributing guidelines

I've found a bug and checked that ...

Description

Expected behaviour

Actual behaviour

What happens on the wire

Why the connections go idle

Two things that turn a slow upload into a hard failure

Reproduction

Expected

Suggestions

Buildx version

Docker info

Builders list

Configuration

Testing from Perth NBN Connection.

Outcome

Test process

Build logs

Additional info

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions