0

Problem

Git hangs while writing objects for a 81 kB repo over HTTPS (SSH is not allowed). The specific error message from Nginx's access.log is as follow:

POST /Hello-World/.git/git-receive-pack HTTP/1.1" 504 183 "-" "git/2.28.0.windows.1"

Background

I host Nginx and Git servers that are inside a mounted Debian image. Important to note: my www for all my website files are mounted separately as read-only. Also my Git root folder is inside www (i.e., www/git).

sudo mount -o loop,offset=1 raspbian-stretch-lite.img /mnt/
sudo mount --bind www/ /mnt/usr/share/nginx/www/
sudo mount -o bind,remount,ro /mnt/usr/share/nginx/www

My Nginx conf is as follows:

server {
  listen 443 ssl;
  server_name git.domain.com;
  ssl_certificate fullchain.pem;
  ssl_certificate_key privkey.pem;

  # fast clone static files
  location ~ \.git/objects/(?:[0-9a-f]+/[0-9a-f]+|pack/pack-[0-9a-f]+\.(?:pack|idx))$ {
    root /usr/share/nginx/www/git;
    limit_except GET HEAD {
      allow all;
    }
  }

  location ~ ^.*\.git/(HEAD|info/refs|objects/info/.*|git-(upload|receive)-pack) {
    root /usr/share/nginx/www/git;
    auth_basic "Restricted";
    auth_basic_user_file /usr/share/nginx/www/auth;
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME /usr/lib/git-core/git-http-backend;
    fastcgi_param GIT_HTTP_EXPORT_ALL "";
    fastcgi_param GIT_PROJECT_ROOT $realpath_root;
    fastcgi_param REMOTE_USER $remote_user;
    fastcgi_param PATH_INFO $uri;
    fastcgi_param unix:/var/fcgiwrap.socket;
  }

  location /gitweb.cgi {
    root /usr/share/nginx/www/gitweb;
    include fastcgi_params:
    gzip off;
    fastcgi_param SCRIPT_NAME $uri:
    fastcgi_param GITWEB_CONFIG $realpath_root/gitweb.conf;
    fastcgi_pass unix:/var/fcgiwrap.socket;
  }

  location / {
    root /usr/share/nginx/www/gitweb;
    index gitweb.cgi;
  }
}

If I visit https://git.domain.com from browser, I get a Gitweb GUI. I clone with git clone https://git.domain.com/Hello-World/.git and all is well. But when I try to push using git push, it hangs at writing object step, eventually giving me a gateway timeout error. Interestingly, every time this happens, both https://git.domain.com Gitweb page and Git backend shut down, forcing me to kill and restart my fcgiwrap process.

My guess

I've tried git config --global http.postBuffer 524288000 to no avail. Since I'm authenticating (by typing my username and password), git-receive-pack should be allowed, right?

I'm assuming this has something to do with git folder not having write permission? Perhaps my Nginx config's regex is not correct?

Edits:

Normally errors from Git would go to stderr, which is usually written into a log by the web server. Is there such a log on your system, and if so, what does it contain?

The error I get in my command prompt from client-side is as follows:

(base) C:\Users\pairwiseseq\Hello-World>git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Writing objects: 100% (3/3), 258 bytes | 258.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504
fatal: the remote end hung up unexpectedly
fatal: the remote end hung up unexpectedly
Everything up-to-date

Also from Nginx's error.log:

*9 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 123.456.7.890, server: git.domain.com, request: "POST /Hello-World/.git/git-receive-pack HTTP/1.1", upstream: "fastcgi://unix:/var/fcgiwrap.socket", host: "git.domain.com"

strace for fcgiwrap process:

read(3, "\1\1\0\1\0\10\0\0\0\1\1\0\0\0\0\0\1\4\0\1\0041\7\0\f\0QUERY_STRING\16\4REQUEST_METHODPOST\f&CONTENT_TYPEapplication/x-git-receive-pack-request\16\3CONTENT_LENGTH452\v\"SCRIPT_NAME/Hello-World/.git/git-receive-pack\v\"REQUEST_URI/Hello-World/.git/git-receive-pack\f\"DOCUMENT_URI/Hello-World/.git/git-receive-pack\r\30DOCUMENT_ROOT/usr/share/nginx/www/git\17\10SERVER_PROTOCOLHTTP/1.1\16\5REQUEST_SCHEMEhttps\5\2HTTPSon\21\7GATEWAY_INTERFACECGI/1.1\17\fSERVER_SOFTWAREnginx/1.10.3\v\rREMOTE_ADDR123.456.7.890\v\5REMOTE_PORT39118\v\fSERVER_ADDR123.456.7.12\v\3SERVER_PORT443\v\20SERVER_NAMEgit.domain.com\17\3REDIRECT_STATUS200\17\"SCRIPT_FILENAME/usr/lib/git-core/git-http-backend\23\0GIT_HTTP_EXPORT_ALL\20\30GIT_PROJECT_ROOT/usr/share/nginx/www/git\v\10REMOTE_USERusername\t\"PATH_INFO/Hello-World/.git/git-receive-pack\24\4HTTP_ACCEPT_ENCODINGgzip\v\10HTTP_PRAGMAno-cache\17\31HTTP_USER_AGENTJGit/5.1.2.201810061102-r\22*HTTP_AUTHORIZATIONBasic cGc5eno0c2g6WWw5V1RpdnVrWlE2QTRNRw==\21&HTTP_CONTENT_TYPEapplication/x-git-receive-pack-request\v%HTTP_ACCEPTapplication/x-git-receive-pack-result\25\4HTTP_CONTENT_ENCODINGgzip\23\3HTTP_CONTENT_LENGTH452\t\20HTTP_HOSTgit.domain.com\17\nHTTP_CONNECTIONKeep-Alive\0\0\0\0\0\0\0\1\4\0\1\0\0\0\0\1\5\0\1\1\304\4\0\37\213\10\0\0\0\0\0\0\0\1\255\1R\37600b97fd1a60b01f91b314f59955a4e4d4e80d8edf11d f1a5d26dcb87f31b127cd397894aae43f86f362c refs/heads/master\0report-status delete-refs ofs-delta side-band-64k agent=JGit/5.1.2.201810061102-r0000PACK\0\0\0\2\0\0\0\3\234\fx\234\225\313\301\r\3020\f@\321{\246\310\2\225\354&v\32\t!\6\340\306\4I\354\10\16\220*r\367\247\214\300\345\35\276\364m\252zT&\26\222\36\326\232\2\244J\25\267\225\2651\"K<\273f\255\215\334^\246~\314\247.X\30*`\317X\3\306N9\23\225\250Q\242n \233JG\24W\16{\216\351m\234\\~\336\36\307\354\245\351r/\273\215}\tW\217\fk\f\t3\373\5\10\300\265\361~\277\314\364\337\317}\1q~<[\242\2x\234340031Q\10rut\361ue\210\233\301z.Q\341Ks\234\307\375\0O\311\233\5\355N\241:\0\257\246\v\211=x\234\363H\315\311\311WHLO\314\314S\344\2\0 d\4@\4{\334\"\301R\v\30\337\271\275\n\203\3349{D\255S;9\301\247\200\255\1\0\0\0\0\0\0\1\5\0\1\0\0\0\0", 8192) = 1584
write(5, "\37\213\10\0\0\0\0\0\0\0\1\255\1R\37600b97fd1a60b01f91b314f59955a4e4d4e80d8edf11d f1a5d26dcb87f31b127cd397894aae43f86f362c refs/heads/master\0report-status delete-refs ofs-delta side-band-64k agent=JGit/5.1.2.201810061102-r0000PACK\0\0\0\2\0\0\0\3\234\fx\234\225\313\301\r\3020\f@\321{\246\310\2\225\354&v\32\t!\6\340\306\4I\354\10\16\220*r\367\247\214\300\345\35\276\364m\252zT&\26\222\36\326\232\2\244J\25\267\225\2651\"K<\273f\255\215\334^\246~\314\247.X\30*`\317X\3\306N9\23\225\250Q\242n \233JG\24W\16{\216\351m\234\\~\336\36\307\354\245\351r/\273\215}\tW\217\fk\f\t3\373\5\10\300\265\361~\277\314\364\337\317}\1q~<[\242\2x\234340031Q\10rut\361ue\210\233\301z.Q\341Ks\234\307\375\0O\311\233\5\355N\241:\0\257\246\v\211=x\234\363H\315\311\311WHLO\314\314S\344\2\0 d\4@\4{\334\"\301R\v\30\337\271\275\n\203\3349{D\255S;9\301\247\200\255\1\0\0", 452) = 452
read(6, "Expires: Fri, 01 Jan 1980 00:00:00 GMT\r\nPragma: no-cache\r\nCache-Control: no-cache, max-age=0, must-revalidate\r\nContent-Type: application/x-git-receive-pack-result\r\n\r\n", 4096) = 165

After this, fcgiwrap just completely hangs. I can neither clone nor push at this point and have to spawn a new fcgiwrap process.

pairwiseseq
  • 173
  • 1
  • 1
  • 8
  • Please add the message from nginx `error.log` to your question. – Tero Kilkanen Oct 10 '20 at 09:16
  • @TeroKilkanen, I've added the `error.log` warning to my updated post. Thanks! – pairwiseseq Oct 11 '20 at 16:30
  • This means that your Git application times out during the request. You can find more info about the timeour reason in that application's log file. – Tero Kilkanen Oct 11 '20 at 17:05
  • @TeroKilkanen, correct me if I'm wrong, I don't think Git server provides logging for the entire server. Or maybe you're referring to `fastcgi`? But this is only a protocol that also doesn't provide logging. Ideally I'd like to refrain from increasing timeout limit as the default 60sec is plenty of time to push a `<10kB` repo on 50 Mbps connection. – pairwiseseq Oct 11 '20 at 17:16
  • @TeroKilkanen, I've included some logs from `strace fcgiwrap`. Would you mind taking a look? – pairwiseseq Oct 11 '20 at 17:38
  • Whatever entity generates the response to the request is taking too long. That is why nginx stops the request. Most likely it is git that causes the timeout, so you need to figure out why git action takes so long. – Tero Kilkanen Oct 11 '20 at 21:03

0 Answers0