The Real Fix for Flaky Docker Buildx CI Downloads

Last month our GitLab pipeline started failing at the strangest point. Not during the actual Docker build. Not during tests. It kept dying during the setup step, the part that just downloads and installs the Buildx plugin before any real work happens.

The error looked harmless at first. A curl timeout here, a runner_system_failure there. But it kept happening across two separate frontend projects, and every failure meant a deploy that had nothing wrong with its code got blocked anyway. I spent a week patching this the obvious way, with retries. Then I realized retries were treating the symptom, not the disease.

Key Takeaways

Docker Buildx setup can fail in CI even when your code is fine, because the pipeline redownloads the Buildx binary and rebuilds image layers from scratch on every run. Curl retry flags, per project plugin caching, and package manager installs reduce how often that download fails, but they do not remove the dependency on network reliability. The structural fix is registry based build caching with docker buildx build --cache-from and --cache-to, which reuses both the Buildx setup and prior image layers instead of refetching everything each run. Use retries as defense in depth, not as your primary strategy.

Why Does Docker Buildx Fail During CI Setup, Not During the Build?

Buildx is not always preinstalled on GitLab runners, especially lightweight Alpine based ones. Most teams install it manually in a before_script step, usually by curling the binary straight from GitHub, chmod plus x it, and registering it as a Docker CLI plugin.

That download happens on every single pipeline run, cold, over whatever network path the runner has that day. If the runner has a brief network blip, or GitHub throttles the request, or the runner itself hits a GitLab reported runner_system_failure, the job fails before a single line of your Dockerfile has been processed. Your code did nothing wrong. The infrastructure just had a bad five seconds.

Also Read: Why Docker Buildx Changed My CI/CD Game Forever

Is Adding curl Retry Flags Enough to Fix Buildx Download Timeouts?

No, curl retry flags reduce the failure rate but do not eliminate the underlying fragility. Adding --retry 3 --retry-delay 5 --connect-timeout 10 to the buildx download command is the fastest fix you can ship, and it is genuinely worth doing. It turns a single network blip from a hard pipeline failure into a silent retry.

Here is what that looks like in a GitLab CI before_script.

before_script:
    - |
        ARCH=${CI_RUNNER_EXECUTABLE_ARCH#*/}
        BUILDX_URL="https://github.com/docker/buildx/releases/latest/download/buildx-linux-$ARCH"
        mkdir -vp ~/.docker/cli-plugins/
        curl --retry 3 --retry-delay 5 --connect-timeout 10 \
             --silent -L --output ~/.docker/cli-plugins/docker-buildx "$BUILDX_URL"
        chmod a+x ~/.docker/cli-plugins/docker-buildx

This buys you resilience against short blips. It does nothing against sustained network degradation, GitHub rate limiting, or a runner that simply cannot reach the outside internet for a minute. It is a patch on the symptom, applied at the exact moment your pipeline is most exposed, cold, with no cache, no fallback.

Should You Cache the Buildx Binary or Install It via Package Manager?

Both, and package manager installs should come first when available. Caching the downloaded plugin binary between pipeline runs, keyed by a GitLab CI cache key, avoids re-fetching the same bytes from GitHub every single time. That alone cuts most of the network exposure, since the binary rarely changes between runs.

Where it exists, installing docker-buildx through the OS package manager is even better, because it removes the download step entirely. On Alpine based runners this is as simple as:

before_script:
    - apk add --no-cache docker-buildx

One line, no curl, no chmod, no GitHub dependency at build time. If your runner image supports it, this should be your default, with the curl and cache approach as a fallback for images where the package is not available.

What Is Registry Based Build Caching and Why Does It Fix the Root Cause?

Registry based build caching is a Docker Buildx feature that stores build layer metadata and blobs in a container registry so future builds can reuse them instead of rebuilding from scratch. This is the part that actually attacks the root cause instead of the symptom.

Retry flags and plugin caching only address the setup step, the few seconds before your build even starts. They do nothing for the build itself, which is where most CI pipelines actually spend their time and their network budget, pulling base images, reinstalling dependencies, and recompiling layers that have not changed since the last run.

Registry caching changes what the pipeline needs to fetch in the first place. Instead of rebuilding every layer on every run, Buildx checks the registry for a matching cache entry and reuses it. Less work means less exposure to a flaky network, because there is simply less to download and less to rebuild.

How Do You Set Up docker buildx build with Registry Cache in GitLab CI?

You add --cache-from and --cache-to flags pointing at a dedicated cache tag in your registry, alongside your normal build and push flags. Here is a complete before and after.

Before, a typical build step with no layer caching strategy:

build_job:
    stage: build
    script:
        - docker buildx build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --push .

After, with registry based caching added:

build_job:
    stage: build
    script:
        - docker buildx build
          --cache-from type=registry,ref=$CI_REGISTRY_IMAGE:cache
          --cache-to type=registry,ref=$CI_REGISTRY_IMAGE:cache,mode=max
          -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
          --push .

The mode=max setting tells Buildx to export every layer to the cache, not just the final stage, which matters most for multi stage Dockerfiles. If your registry does not support the newer cache manifest format, the simpler inline cache variant works as a fallback.

- docker buildx build --cache-from type=registry,ref=$CI_REGISTRY_IMAGE:latest
  --build-arg BUILDKIT_INLINE_CACHE=1
  -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --push .

Also Read: Fixing Next.js Docker Build OOM Kills in CI

Do You Still Need Retry Flags If You Use Registry Caching?

Yes, keep them. Registry caching reduces how much your pipeline depends on network reliability, but it does not remove Buildx setup from the pipeline entirely, and it does not make the registry itself immune to blips. Treat curl retry flags, per project plugin caching, and package manager installs as defense in depth underneath the structural fix, not as a replacement for it.

One caveat worth stating plainly. Do not cache forever. Rebuild without cache on a schedule, weekly or biweekly depending on how often your base images ship security patches, so your images still pick up OS level updates instead of silently reusing stale layers indefinitely.

Frequently Asked Questions

Why does my GitLab CI pipeline fail before the Docker build even starts? This usually means the Buildx plugin install step failed, not the build itself. A curl timeout downloading the Buildx binary or a runner_system_failure from GitLab’s infrastructure will kill the job before your Dockerfile is ever processed.

What does mode=max mean in docker buildx build —cache-to? It tells Buildx to export cache data for every stage of a multi stage build, not just the final image. Without it, intermediate build stages are not cached, which limits how much time you actually save on rebuilds.

Is inline cache the same as registry cache? Inline cache is a simpler form of registry caching that embeds cache metadata directly in the pushed image, using the BUILDKIT_INLINE_CACHE build argument. It works with any registry but supports fewer cache export options than the dedicated type=registry cache backend.

Should I switch from curl install to apk add docker-buildx? Yes, if your CI runner image is Alpine based and the package is available. It removes the flaky download step entirely and is more reliable than any amount of curl retry tuning.

How often should I rebuild my Docker image without cache? A common practice is weekly or biweekly, or tied to your base image’s security patch cadence. This keeps you from indefinitely reusing layers that are missing OS level security updates.

Flaky CI is annoying enough that it is tempting to just retry your way out of it. Retries are fine as a safety net. But if your Buildx setup step keeps failing, or your builds keep taking longer than they should, the fix that actually holds up is registry based caching, not another retry flag.