| title | GitLab CI — No Space Left on Device | |||||
|---|---|---|---|---|---|---|
| slug | gitlab-no-space-left-on-device | |||||
| technologies |
|
|||||
| severity | high | |||||
| tags |
|
|||||
| related |
|
|||||
| last_reviewed | 2026-06-27 |
$ docker build -t app:ci .
write /var/lib/docker/tmp/buildkit-mount123/layer.tar: no space left on device
ERROR: Job failed: exit code 1
fatal: write error: No space left on device
error: failed to write object
tar: write error: No space left on device
No space left on device (errno ENOSPC) means a write failed because the
underlying filesystem — or its inodes — filled up. On GitLab runners this almost
always means the runner host's disk is full: accumulated build directories,
Docker images/layers/volumes, the cache, or job artifacts have not been cleaned
up. It can also be the DinD daemon's storage filling during docker build.
The job that happens to be running when the disk hits 100% is the one that fails,
even if it didn't cause the buildup.
- gitlab (GitLab Runner host, Docker storage)
high — once a runner's disk is full, every job on it fails until space is reclaimed, taking the runner out of service.
- Accumulated Docker artifacts: dangling images, stopped containers, and unused volumes never pruned on the runner host.
- Stale build directories under the runner's
builds_dirfrom previous jobs. - Large or unbounded cache and artifacts growing over time.
- A single job writing a huge file/layer (big image build, large test output) that exhausts a small volume.
- Inode exhaustion from millions of tiny files even when bytes are free.
The filesystem returns ENOSPC when there are no free blocks (or no free inodes)
to satisfy a write. GitLab Runner keeps per-job build directories, caches, and —
for Docker/DinD executors — image layers and volumes on the host. Without pruning,
these grow until the volume fills. Because the failure is whichever write happens
to cross the threshold, the error appears in an unrelated job (a git fetch, a
tar, a layer write), which is why it is easy to misattribute. Checking df and
df -i on the runner host immediately confirms the real cause.
# Free space AND inodes on the runner host (both can trigger ENOSPC)
df -h
df -i
# Where the runner stores builds/cache (from config)
grep -nE 'builds_dir|cache_dir|\[runners\]' /etc/gitlab-runner/config.toml
# Biggest consumers under Docker and the runner build dir
docker system df -v 2>/dev/null
du -xhd1 /var/lib/docker /home/gitlab-runner/builds 2>/dev/null | sort -h | tail
# Confirm the message in recent runner logs
journalctl -u gitlab-runner --since "30 min ago" --no-pager | grep -i 'no space'# Disk full:
$ df -h /var/lib/docker
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p1 100G 100G 0 100% /
# Or inodes full while bytes look fine:
$ df -i
/dev/nvme0n1p1 6.5M 6.5M 0 100% /
-
Reclaim Docker space (safe, removes only unused objects):
docker system prune -af --volumes # dangling images, stopped containers, unused volumes -
Remove stale runner build directories (only when no jobs are running):
sudo find /home/gitlab-runner/builds -mindepth 1 -maxdepth 1 -type d -mtime +2 -exec rm -rf {} + -
Bound cache/artifacts: set
expire_inon artifacts and cap cache size; clear old runner cache directories. -
Grow the volume if the runner is legitimately undersized, or move
/var/lib/dockerto a larger disk. -
For inode exhaustion, delete the directories with the most files (often old
node_modules/cache) rather than chasing bytes.
df -h && df -i # Use% well below 100 on both
# Re-run a pipeline; the previously failing write step now completes, exit 0.- Schedule periodic
docker system prune -afand build-dir cleanup on every runner (cron/systemd timer). - Set
expire_inon allartifacts:and a sane cache policy. - Alert on runner disk and inode usage crossing ~80%, before jobs start failing.
- GitLab Runner — Prepare Environment: Exit Status 1
- GitLab DinD — Cannot Connect to the Docker Daemon
gitlab · runner · disk · storage · production