In Sonatype Nexus Docker proxy, how do I debug "Bad request" error?

Question

I've been following the instructions to set up a Proxy Repository for Docker. I am intending to set up a proxy for Docker hub, that is for https://index.docker.io/.

My setup details on Nexus

• Nexus version 3.36.0-01
• Installed via docker-compose
• Generated/installed self-signed cert
• Using built-in https/jetty, NOT reverse proxy
• http listening on port 80
• https listening on port 443

My setup details on Nexus docker proxy repo

• Configured for https, port 8443
• Proxy remote storage: https://registry-1.docker.io
• Proxy docker index: "use docker hub", pre-filled as https://index.docker.io/
• Allowing anonymous docker pull
• Enabled Docker Bearer Token Realm
• Enabled docker v1 API
• Enabled foreign layer caching

My setup details on Ubuntu docker client

• Trusted self-signed cert in /etc/docker/certs.d
• Trusted self-signed cert in /usr/local/share/ca-certificates + update-ca-certificates
• Enabled Docker daemon debugging in /etc/docker/daemon.json
• Enabled Docker proxy via httpsProxy in /home/myuser/.docker/config.json
• Enabled Docker proxy via httpsProxy in /etc/systemd/system/docker.service.d/https-proxy.conf, reloaded/restarted Docker daemon

My test from the client

• docker pull hello-world:latest
• returns error Error response from daemon: Get https://registry-1.docker.io/v2/: Bad Request
• In debug logs:

Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.190545462Z" level=debug msg="Calling HEAD /_ping"
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.190878019Z" level=debug msg="Calling GET /v1.40/info"
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.213218413Z" level=debug msg="Calling POST /v1.40/images/create?fromImage=hello-world&tag=latest"
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.213290250Z" level=debug msg="Trying to pull hello-world from https://registry-1.docker.io v2"
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.234803592Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: Bad Request"
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.234865780Z" level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: Bad Request"
Nov 17 22:53:17 myclient dockerd[20160]: time="2021-11-17T22:53:17.234976364Z" level=error msg="Handler for POST /v1.40/images/create returned error: Get https://registry-1.docker.io/v2/: Bad Request"

Next Steps

I'm watching the logs on the server while this is happening. It shows no errors. However the client side seems to indicate the request is partly working.

I tried increasing org.apache.http.wire to DEBUG as per this other SO question/answer, but that also showed nothing.

How do I continue debugging?

Answer 1

If you examine the docker output you’ll notice it isn’t going to nexus, it is making the request to https://registry-1.docker.io. To pull from Nexus you need to prepend the host and port to the pull request.

docker pull hostname:8443/hello-world:latest

There isn’t any way in docker to have it default to a private registry btw, so you’ll always need to prepend host:port.