6

I'm running into an issue with using GIT to connect to a GIT server that is protected by client certificates. I'm trying to use the Azure Pipeline Agent to run jobs on a brand new Azure VM.

I'm connecting to an on-premise TFS instance that is secured to the internet via client certificates.

With some trial and error, I got the certificates int the right shape to connect the agent up to the server. Using OpenSSL like this 

openssl pkcs12 -in cert.pfx -cacerts -nokeys -out ca.pem -passin pass:
openssl pkcs12 -in cert.pfx -clcerts -nokeys -out clientcert.pem -passin pass:
openssl pkcs12 -in cert.pfx -clcerts -nocerts -out clientcert-key-pass.pem -passin pass: -passout pass:$pfxpassword
openssl pkcs12 -in cert.pfx -out temp.pem -nodes -passin pass:
openssl pkcs12 -export -out cert-secure.pfx -in temp.pem -passout pass:$pfxpassword

Thats where the fun ends....

For the background, I'm running on Ubuntu 18.04 and have tried GIT 2.17 and 2.22.

When I run curl on the machine with the same set of certificates it correctly gets past the client certificate security on the server and lets me access the underlying service which returns a 401 which is expected.

However, when using the same certificates from GIT I consistently get a 403 error and the server rejects the connection due to issues with the client certs.

The command I am running for GIT is

GIT_CURL_VERBOSE=1 GIT_TRACE=1 git -c http.sslcainfo="/home/scott/ca.pem" -c http.sslcert="/home/scott/clientcert.pem" -c http.sslkey="/home/scott/clientcert-key-pass.pem" -c http.sslCertPasswordProtected=true fetch --force --tags --prune --progress --no-recurse-submodules origin

The result of this is (redacted urls)

07:04:43.587108 git.c:439               trace: built-in: git fetch --force --tags --prune --progress --no-recurse-submodules origin
07:04:43.587518 run-command.c:663       trace: run_command: GIT_DIR=.git git-remote-https origin https://*****
Password for 'cert:////home/scott/clientcert.pem':
* Couldn't find host **** in the .netrc file; using defaults
*   Trying 3.248.79.34...
* TCP_NODELAY set
* Connected to **** (3.248.79.34) port 443 (#0)
* found 3 certificates in /home/scott/ca.crt
* found 402 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
*        server certificate verification SKIPPED
*        server certificate status verification SKIPPED
*        common name: *** (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: **** - redacted
*        start date: Mon, 08 Apr 2019 00:00:00 GMT
*        expire date: Tue, 07 Apr 2020 12:00:00 GMT
*        issuer: C=US,O=DigiCert Inc,CN=DigiCert SHA2 Secure Server CA
*        compression: NULL
* ALPN, server accepted to use http/1.1
> GET /****/info/refs?service=git-upload-pack HTTP/1.1
Host: ****.com
User-Agent: git/2.22.0
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: C, *;q=0.9
Pragma: no-cache

< HTTP/1.1 403 Forbidden
< Content-Type: text/html
< Server: Microsoft-IIS/10.0
< Date: Tue, 13 Aug 2019 07:04:47 GMT
< Content-Length: 1300
<
* Connection #0 to host ****.com left intact
fatal: unable to access 'https://*****': The requested URL returned error: 403

It seems like GIT is skipping the client auth for some reason

The equivalent curl command is

curl -v --http1.0 --cacert /home/scott/ca.pem --key /home/scott/clientcert-key-pass.pem --cert /home/scott/clientcert.pem https://****.com/tfs/****/refs?service=git-upload-pack

and this results in success and can get past the client auth on the webserver and connect to the underlying service

*   Trying 3.248.79.34...
* TCP_NODELAY set
* Connected to tfsemea1.ta.philips.com (3.248.79.34) port 443 (#0)
* ALPN, offering http/1.1
Enter PEM pass phrase:
* successfully set certificate verify locations:
*   CAfile: /home/scott/ca.pem
  CApath: /etc/ssl/certs
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: ******
*  start date: Apr  8 00:00:00 2019 GMT
*  expire date: Apr  7 12:00:00 2020 GMT
*  subjectAltName: host "***.com" matched cert's "***.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
> GET /***/info/refs?service=git-upload-pack HTTP/1.0
> Host: ***.com
> User-Agent: curl/7.58.0
> Accept: */*
>
* TLSv1.2 (IN), TLS handshake, Hello request (0):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
< HTTP/1.1 401 Unauthorized
< Content-Type: text/html; charset=utf-8
< Server: Microsoft-IIS/10.0
< P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
< WWW-Authenticate: Bearer
< WWW-Authenticate: Basic realm="https://***.com/tfs"
< WWW-Authenticate: Negotiate
< WWW-Authenticate: NTLM
< X-TFS-ProcessId: d2b304b0-a0ef-48b9-89b8-3281cb42c26d
< ActivityId: 07d4f17f-3940-4006-aab5-5c7afa57e84a
< X-TFS-Session: 07d4f17f-3940-4006-aab5-5c7afa57e84a
< X-VSS-E2EID: 07d4f17f-3940-4006-aab5-5c7afa57e84a
< X-FRAME-OPTIONS: SAMEORIGIN
< X-TFS-SoapException: %3C%3Fxml%20version%3D%221.0%22%20encoding%3D%22utf-8%22%3F%3E%3Csoap%3AEnvelope%20xmlns%3Asoap%3D%22http%3A%2F%2Fwww.w3.org%2F2003%2F05%2Fsoap-envelope%22%3E%3Csoap%3ABody%3E%3Csoap%3AFault%3E%3Csoap%3ACode%3E%3Csoap%3AValue%3Esoap%3AReceiver%3C%2Fsoap%3AValue%3E%3Csoap%3ASubcode%3E%3Csoap%3AValue%3EUnauthorizedRequestException%3C%2Fsoap%3AValue%3E%3C%2Fsoap%3ASubcode%3E%3C%2Fsoap%3ACode%3E%3Csoap%3AReason%3E%3Csoap%3AText%20xml%3Alang%3D%22en%22%3ETF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.%3C%2Fsoap%3AText%3E%3C%2Fsoap%3AReason%3E%3C%2Fsoap%3AFault%3E%3C%2Fsoap%3ABody%3E%3C%2Fsoap%3AEnvelope%3E
< X-TFS-ServiceError: TF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.
< X-Powered-By: ASP.NET
< Lfs-Authenticate: NTLM
< X-Content-Type-Options: nosniff
< X-Powered-By: ARR/3.0
< Date: Tue, 13 Aug 2019 07:58:27 GMT
< Connection: close
< Content-Length: 20158

At @VonC suggestion I tried without using the same key, without a password, using the following command

GIT_CURL_VERBOSE=1 GIT_TRACE=1 git -c http.sslcainfo="/home/scott/ca.pem" -c http.sslcert="/home/scott/clientcert.pem" -c http.sslkey="/home/scott/nopass.pem" fetch --force --tags --prune --progress --no-recurse-submodules origin

I removed the password using openssl rsa -in ~/clientcert-key-pass.pem -out ~/nopass.pem

It generated a very similar output 

15:22:57.671901 git.c:440               trace: built-in: git fetch --force --tags --prune --progress --no-recurse-submodules origin
15:22:57.672253 run-command.c:663       trace: run_command: GIT_DIR=.git git-remote-https origin https://****
* Couldn't find host **** in the .netrc file; using defaults
*   Trying 52.18.21.126...
* TCP_NODELAY set
* Connected to **** (52.18.21.126) port 443 (#0)
* found 3 certificates in /home/scott/ca.pem
* found 402 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
*        server certificate verification SKIPPED
*        server certificate status verification SKIPPED
*        common name: **** (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: ****
*        start date: Thu, 29 Aug 2019 00:00:00 GMT
*        expire date: Tue, 07 Apr 2020 12:00:00 GMT
*        issuer: C=US,O=DigiCert Inc,CN=DigiCert SHA2 Secure Server CA
*        compression: NULL
* ALPN, server accepted to use http/1.1
> GET /tfs/***/info/refs?service=git-upload-pack HTTP/1.1
Host: ****
User-Agent: git/2.23.0
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: C, *;q=0.9
Pragma: no-cache

< HTTP/1.1 403 Forbidden
< Content-Type: text/html
< Server: Microsoft-IIS/10.0
< Date: Sun, 22 Sep 2019 13:22:56 GMT
< Content-Length: 1300
<
* Connection #0 to host **** left intact
fatal: unable to access 'https://****/': The requested URL returned error: 403

The only difference in the output is the prompt for the password

Password for 'cert:////home/scott/clientcert.pem':

Running git config -l produces the following effective config from within the folder I am using

core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
remote.origin.url=https://****
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
gc.auto=0

Banging my head on this for some time now and am starting to think that something is broken somewhere and that GIT is not starting the client certificate handshake process you can see in curl and that it is just not presenting it and therefore getting the 403.

Am I missing something?

update:

403 - means the webserver is rejecting the request due to client cert issues 401 - means the underlying service (TFS) is rejecting the request due to authentication issues (which is a separate issue)

Chaurasia
  • 494
  • 1
  • 6
  • 22
ScottGuymer
  • 1,885
  • 1
  • 17
  • 22
  • Does setting `http.sslBackend` to `openssl` change anything? – bk2204 Sep 24 '19 at 00:37
  • I had already tried this. The default (and only documented option) for linux is openssl. – ScottGuymer Sep 24 '19 at 07:07
  • The 403 error indicates ether that the git server does not grant you read access to this repository, or that the web server does not grant acces to the git-http-backend CGI program. – user803422 Sep 25 '19 at 09:44

3 Answers3

0

http.sslCertPasswordProtected: every time I had to use certificates, I always used a key from which I removed the password.

For testing, you should try and see if the git fetch command would work better with a key certificate not requiring a password.

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • I tried this (and just tried it again to verify) and no dice. Any ideas if there are any restrictions on what size and types of keys and certs are usable? It looks like it is connecting to the server and just not presenting the cert for some reason. – ScottGuymer Sep 19 '19 at 14:32
  • @ScottGuymer could you edit the question with the git settings and curl log, illustrating that failed connection, *with* a certificate without password? – VonC Sep 19 '19 at 16:16
0

Reading the doc, i noticed the following configuration parameter:

http.sslTry

Attempt to use AUTH SSL/TLS and encrypted data transfers when connecting via regular FTP protocol. This might be needed if the FTP server requires it for security reasons or you wish to connect securely whenever remote FTP server supports it. Default is false since it might trigger certificate verification errors on misconfigured servers.

I don't understand well why it speaks about FTP (maybe bad copy/paste ?) since the configuration parameter is related to http ... but maybe you should try this.

binarym
  • 357
  • 2
  • 10
-1

I am not sure your curl is working much better than git, as you get an error as well:

HTTP/1.1 401 Unauthorized

Please clarify why you consider that curl succeeded.

What I see is that in both cases (git and curl), the server accepts the client during the TLS handshake, and the request is refused afterwards by the backend web service. (if the TLS handshake had failed, then you would have never seen the > GET ... parts)

How does the backend web service decide to grant access or not? Does it use the client certificate information?

user803422
  • 2,636
  • 2
  • 18
  • 36
  • I will update the question but 403 - means the web server is rejecting the request due to client cert issues 401 - means the service is rejecting the request due to authentication issues (which is a separate issue) – ScottGuymer Sep 25 '19 at 07:13
  • I think i see what you are saying now. That the client certificate negotiation is completing but for some reason the web server it rejecting the certificate as not authorised. Its the exact same cert as in the curl version so im not sure why this would happen. – ScottGuymer Sep 25 '19 at 07:19
  • 1
    This would be better as a comment, it is not an answer. – istepaniuk Sep 25 '19 at 08:28