Resolved. The BuildKit cache is capped at 8 GB on every build host, web-04 has joined as a fourth build host, and deployd 0.4.3 now waits up to 5 minutes for nodes to pull the image during rollout instead of 90 seconds. A follow-up bug where every deploy of a project was rolled back immediately after one failed rollout is fixed as well. New builds complete again; please restart any failed builds once.

The BuildKit cache has been cleared on all three build hosts, and the clean-up rule now also caps the cache at 8 GB per host regardless of entry age. web-01 has 15 GB free again. Failed builds can be restarted; we are watching the next builds.

Builds in CitexNode Deploy assigned to the build host web-01 fail with "no space left on device". Cause: the BuildKit cache on web-01 grew past its limit because the clean-up rule only removed entries older than 7 days. Running websites and apps are not affected, only new builds. We are clearing the cache and enforcing a size limit on all build hosts.