error parsing HTTP 404 response body: invalid character '<' looking for beginning of value docker

Question

I am currently trying to configuring nexus3 as private registry for docker images with nginx serving as a reverse proxy . Created 3 repos within nexus namely NexusDockerProxy(docker proxy), NexusDockerHosted(docker hosted,http port:4444) & NexusDockerGroup(docker group,http port:5555) with both the hosted and proxy added.

The nexus is configured with nginx where we are using a self signed certificate and have added the same to nginx configuration file .

server {

    proxy_send_timeout 120;
    proxy_read_timeout 300;
    proxy_buffering    off;
    tcp_nodelay        on;

    server_tokens off;
    client_max_body_size 1G;

    listen 80;
    server_name server908.int.org.com;
    location / {
          rewrite ^(.*) https://server908.int.org.com$1 permanent;
    }
}

server {

    listen 443;
    server_name server908.int.org.com;
    keepalive_timeout 60;
    ssl on;
    ssl_certificate /etc/ssl/certs/orgnexus.crt;
    ssl_certificate_key /etc/ssl/certs/orgnexus.key;
    ssl_ciphers HIGH:!kEDH:!ADH:!MD5:@STRENGTH;
    ssl_session_cache shared:TLSSSL:16m;
    ssl_session_timeout 10m;
    ssl_prefer_server_ciphers on;

    location / {

      proxy_set_header        Host $http_host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto "https";
      proxy_pass              http://server908.int.org.com:8082;
      proxy_read_timeout      90;

    }
}

# correlates to your nexus http connector
server {

    listen 6666;
    server_name server908.int.org.com;
    keepalive_timeout 60;
    ssl on;
    ssl_certificate /etc/ssl/certs/orgnexus.crt;
    ssl_certificate_key /etc/ssl/certs/orgnexus.key;
    ssl_ciphers HIGH:!kEDH:!ADH:!MD5:@STRENGTH;
    ssl_session_cache shared:TLSSSL:16m;
    ssl_session_timeout 10m;
    ssl_prefer_server_ciphers on;
    client_max_body_size 1G;
    chunked_transfer_encoding on;

    location / {

      access_log              /var/log/nginx/docker.log;
      proxy_set_header        Host $http_host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto "https";
      proxy_pass              http://server908.int.org.com:5555;
      proxy_read_timeout      90;

    }
}

We have commented below entries with "/etc/default/docker" file .

http_proxy=http://x.x.x.x:3128
https_proxy=http://x.x.x.x:3128

Login is successful too

[root@server446 ~]$ docker login -u admin -p admin123 server908.int.org.com:6666
Login Succeeded

Search & pull for images too works fine .

[test@server446 ~]$ docker search server908.int.org.com:6666/mac
INDEX      NAME                                                                 DESCRIPTION                                     STARS     OFFICIAL   AUTOMATED
org.com   server908.int.org.com:6666/crate                                    CrateDB is a distributed SQL database hand...   79        [OK]
org.com   server908.int.org.com:6666/maccam912/meanjs                         MEANJS (not MEANIO) starting point for dev...   27                   [OK]
org.com   server908.int.org.com:6666/macadmins/puppetmaster                   Simple puppetmaster based on CentOS 6           25                   [OK]
org.com   server908.int.org.com:6666/macadmins/bsdpy                          Apple NetBoot replacement written in Pytho...   18                   [OK]
org.com   server908.int.org.com:6666/macadmins/reposado                       Host Apple Software Updates from a Docker ...   12                   [OK]
org.com   server908.int.org.com:6666/mackerel/mackerel-agent                                                                  11                   [OK]
org.com   server908.int.org.com:6666/macadmins/munkiwebadmin                  This Docker container runs MunkiWebAdmin. ...   8                    [OK]
org.com   server908.int.org.com:6666/macadmins/postgres                       Postgres that accepts remote connections b...   8                    [OK]
org.com   server908.int.org.com:6666/macropin/strider                         Possibly the best `Dockerfile` for Strider-CD   8                    [OK]
org.com   server908.int.org.com:6666/mace/openvpn-as                          OpenVPN Access-Server with WebUI                7                    [OK]
org.com   server908.int.org.com:6666/frolvlad/alpine-python-machinelearning   Small Docker image with Python Machine Lea...   6                    [OK]
org.com   server908.int.org.com:6666/macadmins/munkireport-php                Docker autobuild repository for https://re...   6                    [OK]
org.com   server908.int.org.com:6666/mace/qbittorrent                         qBittorrent build from source                   6                    [OK]
org.com   server908.int.org.com:6666/macropin/sshd                            (deprecated) Use docker.io/panubo/sshd          5                    [OK]
org.com   server908.int.org.com:6666/macadmins/macnamer                                                                       4                    [OK]
org.com   server908.int.org.com:6666/macadmins/puppetmaster-whdcli            Puppetmaster + WHDCLI for autosigning base...   4                    [OK]
org.com   server908.int.org.com:6666/macadmins/sal                            An image that runs [Sal](https://github.co...   4                    [OK]
org.com   server908.int.org.com:6666/macropin/roundcube                       Docker container for Roundcube webmail          4                    [OK]
org.com   server908.int.org.com:6666/uetchy/machinelearning                   Dockerfile for scientists studying Machine...   4                    [OK]
org.com   server908.int.org.com:6666/clearlinux/machine-learning              Machine Learning Container                      3                    [OK]
org.com   server908.int.org.com:6666/hartator/wayback-machine-downloader      Download an entire archive from the Waybac...   3                    [OK]
org.com   server908.int.org.com:6666/macadmins/whd                            Docker container for SolarWinds WebHelpDesk     3                    [OK]
org.com   server908.int.org.com:6666/macinv/gunicorn-example                  A simple Gunicorn example                       2                    [OK]
org.com   server908.int.org.com:6666/macropin/mailman                         Mailman all in one container                    2                    [OK]
org.com   server908.int.org.com:6666/spacemacs/emacs25                        Spacemacs running on Emacs 25.                  2                    [OK]

[test@server446 ~]$ docker pull server908.int.org.com:6666/macadmins/puppetmaster
Using default tag: latest
Trying to pull repository server908.int.org.com:6666/macadmins/puppetmaster ...
sha256:bf7a53a0ddb6cd0371cfec3fe6131347b5a3712c2cf282400c979ed254a38f67: Pulling from server908.int.org.com:6666/macadmins/puppetmaster
0d399614d56a: Pull complete
75e4c6031170: Pull complete
12a00ed15a39: Pull complete
6fb754ec65e3: Pull complete
9b6cf7664f02: Pull complete
07db30a930a5: Pull complete
ccdd2cf9b303: Pull complete
820dfdfc6db7: Pull complete
78c5dec8ba5d: Pull complete
Digest: sha256:bf7a53a0ddb6cd0371cfec3fe6131347b5a3712c2cf282400c979ed254a38f67
Status: Downloaded newer image for server908.int.org.com:6666/macadmins/puppetmaster:latest

But while pushing the images we get 404 error as below .

[test@server446 ~]$ docker push server908.int.org.com:6666/maven:1
The push refers to a repository [server908.int.org.com:6666/maven]
701925f78142: Layer already exists
78bb4fee972f: Layer already exists
e1300844f726: Layer already exists
bfee0515af91: Preparing
2afcf4c557eb: Preparing
72cfa243711c: Waiting
9bf603e17b04: Waiting
70b22baddf90: Waiting
596ecbaf3ba4: Waiting
445ed6ee6867: Waiting
c59fa6cbcbd9: Waiting
8d4d1ab5ff74: Waiting
error parsing HTTP 404 response body: invalid character '<' looking for beginning of value:

I am really trying to get this to work but now stuck with this last step . Any help to get docker push working would be highly helpful thanks.

I realized that this issue is two years old, but I figured for others stopping by it's still worth an answer. If you figured it out in the meantime: Post your own answer, as I think it'll be better than mine ;) — Worp, Jun 27 '19 at 19:09

score 8 · Answer 1 · answered Jun 27 '19 at 19:04

The error error parsing HTTP 404 response body... is typical for nexus trying to tell you that you're pushing to the group or a proxy repository, when you can only push to a hosted repo.

As far as I can tell, this explanation might fit:

With nginx config being

server {
    listen 6666;
    server_name server908.int.org.com;
...
        proxy_pass              http://server908.int.org.com:5555;
        proxy_read_timeout      90;
...
}

And your push going to this proxy:

docker push server908.int.org.com:6666/maven:1

You would be ultimately be pushing to server908.int.org.com:5555.

With your Nexus config above being

Created 3 repos within nexus namely NexusDockerProxy(docker proxy), NexusDockerHosted(docker hosted,http port:4444) & NexusDockerGroup(docker group,http port:5555) with both the hosted and proxy added

You are pushing to the Docker group. I think that's why it throws the above error.

Configure your nginx-proxy to forward to 4444 instead of 5555 and you should be fine to push.

Sidenote: This makes it necessary though to have two endpoints (one for "reading from the group") and one for pushing to Nexus.

Thus, set up two endpoints in your nginx config (one proxying to 5555 and one proxying 4444). Use the proxy to 4444 in all your Dockerfiles and when pulling from your registry. Use the proxy to 5555 only when pushing.

So I suggest something like this might do the trick:

# Proxy 1, the "pull" proxy. Use this when pulling and in your Dockerfiles/docker-compose files
server {

    listen 6666;
    server_name server908.int.org.com;
    [...ssl config and everything else...]

    location / {

        access_log              /var/log/nginx/docker-pull.log;
        [...header config and other things...]
        proxy_pass              http://server908.int.org.com:5555;
        proxy_read_timeout      90;
    }
}

# Proxy 2, the "push" proxy. Use this endpoint ONLY when pushing images to Nexus.
server {

    listen 7777;
    server_name server908.int.org.com;

    [...ssl config and everything else...]

    location / {

        access_log              /var/log/nginx/docker-push.log;
        [...header config and other things...]
        proxy_pass              http://server908.int.org.com:4444;
        proxy_read_timeout      90;
    }
}

Now I am not a Nexus expert, but this should fix your problem. I am not certain if Nexus has intended this to work via one endpoint (group AND hosted Docker repo via one port), but I remember setting it up with these two endpoints back in the day, one for pushing one for pulling.

Hope it helps!

Alim Azad · Accepted Answer · 2019-04-15T00:54:14.453

Its working now . Configure server port 6666 block with different proxy port to be used . For GET request proxy port should be 5555(i.e. docker-proxy repo) port and other than GET proxy port should be 4444(i.e. docker-hosted repo) port

Commented proxy entries within "/etc/default/docker" file .

Restart Nginx.

Perform Login

Post login a file name "config.json" will be created under ".docker" directory

Search for Images available in docker hub. Pull the images from docker hub via nexus proxy Tag The pulled images Push the image

Pull from Nexus Repo (This should be quick than the pull from docker hub ) Also make sure you add you proxy server certificate within Nexus SSL Certificate Section .

keytool -J-Dhttps.proxyHost=<proxy_hostname> -J-Dhttps.proxyPort=<proxy_port> -printcert -rfc -sslserver <remote_host_name:remote_ssl_port>

Replace and with the HTTP proxy server that Nexus is configured with under Administration -> Server. Replace with one of the remote host and port having the certification problem. You can omit the port if it is the default 443.

You should see at least two entries printed by the above command. Take the last certificate content printed and copy it entirely to your clipboard. This should be your proxy server's certificate, added to the end of the certificate chain.

The copied certificate content should start with -----BEGIN CERTIFICATE----- and end with-----END CERTIFICATE-----.

Then in the Nexus UI, go to Administration -> SSL Certificates and click Add... and choose Paste PEM. Paste the certificate contents to the dialog that opens.

Click Load Certificate. Verify the certificate contents in the next window. Verify the Issuer details listed are from your proxy server certificate. When you are satisfied click Add Certificate.

Hope this helps you .

I have the same use case for Nexus-Docker and I don't have reverse proxy. I see the same issue. This problem seem to be related then not about nginx configuration but something else? — Dimon Buzz, Oct 18 '17 at 22:08

error parsing HTTP 404 response body: invalid character '<' looking for beginning of value docker

2 Answers2