1

I have Nginx 1.16.1 as a reverse proxy for JFrog Artifactory and they are reachable from the external networks via web application firewall. I am trying to get docker client working with this setup. It sends a HEAD request and awaits a Content-Length to check for the existence of a layer. Now I see that Content-Length is not included in the response received by the client. I can examine it by sending the same request using curl that sends docker:

$ curl -H 'User-Agent: docker/19.03.13 go/go1.13.15 git-commit/4484c46d9d kernel/4.19.128-microsoft-standard os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.13 \(linux\))' \
  -H "Authorization: Bearer ${TOKEN}" \
  -H 'Connection: close' \
  -I \
   "https://${ARTIFACTORY_URL}/v2/${IMAGE}/blobs/${DIGEST}"
HTTP/1.1 200 OK
Date: Mon, 09 Nov 2020 14:57:05 GMT
Server: Secure Entry Server
Content-Type: application/octet-stream
Docker-Content-Digest: sha256:[MASKED]
Docker-Distribution-Api-Version: registry/2.0
X-Artifactory-Id: [MASKED]
X-Artifactory-Node-Id: [MASKED]
Set-Cookie: SCDID_S=[MASKED]; path=/; Secure; HttpOnly
Connection: close

However, I see in the access log of Artifactory that it sets this response header. I used tcpdump to see what data is exchanged between Nginx and Artifactory:

HEAD /v2/[MASKED]/blobs/[MASKED] HTTP/1.1
X-JFrog-Override-Base-Url: https://[MASKED]:443
X-Forwarded-Port: 443
X-Forwarded-Proto: https
Host: [MASKED]
X-Forwarded-For: 10.10.40.14
Connection: close
ClientCorrelator: 0rIKeSpqZ9E$
RequestCorrelator: 7f0100-9099-2020.11.09_1457.05.275-001
HSP_CLIENT_ADDR: [MASKED]
Hsp-ListenerUri: https://[MASKED]
HSP_HTTPS_HOST: [MASKED]:443
Accept: */*
Authorization: Bearer [MASKED]
User-Agent: docker/19.03.13 go/go1.13.15 git-commit/4484c46d9d kernel/4.19.128-microsoft-standard os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.13 \(linux\))
HTTPS: on
SSLSessionID: 78ad360e9ea54f5efdb72ea223a63b6cbc7788ae9a1e876620e398040d06182c
SSLSessionTimeLeft: 3600
SSLSessionAge: 0
SSLCipher: ECDHE-RSA-AES128-GCM-SHA256
SSLCipherKeySize: 128
SSLProtocolVersion: TLSv1.2
Via: HTTP/1.1 Secure Entry Server

HTTP/1.1 200 OK
Content-Length: 2529
Content-Type: application/octet-stream
Date: Mon, 09 Nov 2020 14:57:05 GMT
Docker-Content-Digest: [MASKED]
Docker-Distribution-Api-Version: registry/2.0
Server: Artifactory/7.4.1 70401900
X-Artifactory-Id: 5a2dee84b6d80d2f:1f521881:17554c79de4:-8000
X-Artifactory-Node-Id: [MASKED]
Connection: close

The TrafficAnalyzer on the WAF shows that Content-Length in the incoming response from Artifactory is missing. Hence it must be Nginx responsible for removing it.

Now when I connect via VPN to get around the WAF the response looks okay:

Host:           [MASKED]
User-Agent:     docker/19.03.13 go/go1.13.15 ...
Authorization:  Bearer [MASKED]
Connection:     close

Date:                           Fri, 06 Nov 2020 17:13:58 GMT
Content-Type:                   application/octet-stream
Content-Length:                 2529
Docker-Content-Digest:          [MASKED]
Docker-Distribution-Api-Version:registry/2.0
Server:                         Artifactory/7.4.1 70401900
X-Artifactory-Id:               5a2dee84b6d80d2f:1f521881:17554c79de4:-8000
X-Artifactory-Node-Id:          [MASKED]
Connection:                     close

But I also notice, that there are fewer headers set in the request. Is that some additional WAF-header that causes Nginx to remove Content-Length? I don't see anything related to this in Nginx debug log. Any thoughts?

Danny Lo
  • 1,553
  • 4
  • 26
  • 48
  • which WAF is your organization using? it is possible that the WAF is removing the content-length header (for example if the response is chunked or compressed) – Dror Bereznitsky Nov 09 '20 at 20:50
  • Hello Dror, the header is missing in the incoming response on the WAF, before any processing applies. Also, you see that the response is neither chunked nor compressed since it's just an answer to a `HEAD` request. – Danny Lo Nov 10 '20 at 08:57

0 Answers0