Duplicate PUT request during push of BuildKit-built image

[skenmy]

Description

Our upstream container repository provider (Oracle Cloud) reports that they are seeing duplicated requests during the push of an image, and thus are serving a 409 Conflict response on the second request.

The requests occur only during the initial push of a new tag on an image built with buildkit (the default in the version of Docker I am running), and not when the legacy builder is used.

Successive calls to push the same tag work, however new tags on an existing repository exhibit the same behaviour.

We have found that older versions of the docker CLI (we have tested up to Docker Engine v26.0.0) work without this issue.

Pushing to Docker Hub works as expected.

Reproduce

Dockerfile

text.txt is a simple text file with "Hello World" inside

FROM busybox:latest
COPY test.txt .
CMD ["echo", "hello"]

Failing request

1. docker build -t lhr.ocir.io/<redacted>/ocir-test:buildx-test .
   Observe the build successfully complete
2. docker push lhr.ocir.io/<redacted>/ocir-test:buildx-test
   Observe a 409 error: failed commit on ref "manifest-sha256:fcd195ebabd830937ca77cbe21c9ddbe3499c4517d211ca5c3fcd6d9931a8a10": unexpected status from PUT request to https://lhr.ocir.io/v2/<redacted>/ocir-test/manifests/buildx-test: 409 Conflict

Successful request

1. DOCKER_BUILDKIT=0 docker build -t lhr.ocir.io/<redacted>/ocir-test:legacy-test .
   Observe the build successfully complete
2. docker push lhr.ocir.io/<redacted>/ocir-test:legacy-test
   Observe the push successfully complete

Expected behavior

docker push of a BuildKit-built image should not produce duplicate requests during the push of the image to the container registry.

docker version

Client:
 Version:           27.5.1
 API version:       1.47
 Go version:        go1.22.11
 Git commit:        9f9e405
 Built:             Wed Jan 22 13:37:19 2025
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.38.0 (181591)
 Engine:
  Version:          27.5.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.11
  Git commit:       4c9b3b0
  Built:            Wed Jan 22 13:41:25 2025
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.25
  GitCommit:        bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e946
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    27.5.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.7.3
    Path:     /Users/pwilliams/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.20.1-desktop.2
    Path:     /Users/pwilliams/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.32.4-desktop.1
    Path:     /Users/pwilliams/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.38
    Path:     /Users/pwilliams/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Beta) (Docker Inc.)
    Version:  v0.1.4
    Path:     /Users/pwilliams/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/pwilliams/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     /Users/pwilliams/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/pwilliams/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/pwilliams/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/pwilliams/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.16.1
    Path:     /Users/pwilliams/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 17
 Server Version: 27.5.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc version: v1.1.12-0-g51d5e946
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.12.5-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 12
 Total Memory: 7.653GiB
 Name: docker-desktop
 ID: a0e8fab6-f787-4bbe-9402-df707d86743b
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/pwilliams/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Additional Info

No response

[thaJeztah]

Let me transfer this to the buildx repository, as this looks to be an issue with BuildKit and/or buildx (the client used for docker build).

[cdupuis]

@thaJeztah isn't the failing command here docker push as indicated in the report?

[thaJeztah]

Oh! 🙈 You're right; I thought this was a docker build --push.

[skenmy]

Admittedly I myself am unsure where the issue lies - but the issue does seem to be at the point of push, rather than the build itself.

[thaJeztah]

Not sure yet!

If you disable BuildKit, Docker uses the legacy builder, which produces a single-arch image (for the platform you're building for). With BuildKit and the containerd image store enabled, BuildKit produces a multi-platform image ("OCI Image Index") as well as attestations, SBOMS related to the image.

My first thinking there is that it's not unlikely that pushing such an image would try a PUT to make sure the manifest exists before pushing the content related to it (or a similar flow).
At a quick glance, the OCI distribution spec (the specification for OCI registries) does not mention a 409 status for the PUT /v2/<name>/manifests/<reference> endpoint (end-7); https://github.com/opencontainers/distribution-spec/blob/v1.1.1/spec.md#endpoints, and I don't see (at a quick glance) in the "pushing manifests" section, other than 413 being allowed for manifests larger than what the registry wants to accept https://github.com/opencontainers/distribution-spec/blob/v1.1.1/spec.md#pushing-manifests

Possibly the registry in this case is configured for immutable tags, therefore not accepting.

It might be worth trying if building without attestations works around the problem (which would narrow down things);
https://docs.docker.com/build/metadata/attestations/#creating-attestations

[skenmy]

Using --provenance=false with BuildKit does allow the image to be pushed successfully.

BUILDX_NO_DEFAULT_ATTESTATIONS=1 achieves the same effect (no 409 error).

--provenance=true creates an image that causes a 409 on push.