git clone gets hung and gets timedout

Question

I use Bitbucket(git) server and currently the git clone is randomly getting hung. The same repo sometimes gets cloned and sometimes get hung unexpectedly and then gets timed out. The server memory, CPU utilization, disk space is fine. Please advise what could be other areas that needs attention to find the root cause.This is happening on 5 different Jenkins slave so it is not restricted to a single server.

Following is the error log:

ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "/usr/bin/git -c core.askpass=true fetch --tags --progress http://bitbucket.net/project.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: remote: Counting objects: 2506, done.   
stdout: 
stderr: remote: Counting objects: 2506, done.        
remote: Compressing objects:   0% (1/2405)           
remote: Compressing objects:   1% (25/2405)           
remote: Compressing objects:   2% (49/2405)           
remote: Compressing objects:   3% (73/2405)           
remote: Compressing objects:   4% (97/2405)           
remote: Compressing objects:   5% (121/2405)           
remote: Compressing objects:   6% (145/2405)           
remote: Compressing objects:   7% (169/2405)           
remote: Compressing objects:   8% (193/2405)           
remote: Compressing objects:   9% (217/2405)           
remote: Compressing objects:  10% (241/2405)           
remote: Compressing objects:  11% (265/2405)           
remote: Compressing objects:  12% (289/2405)           
remote: Compressing objects:  13% (313/2405)           
remote: Compressing objects:  14% (337/2405)           
remote: Compressing objects:  15% (361/2405)           
remote: Compressing objects:  16% (385/2405)           
remote: Compressing objects:  17% (409/2405)           
remote: Compressing objects:  18% (433/2405)           
remote: Compressing objects:  19% (457/2405)           
remote: Compressing objects:  20% (481/2405)           
remote: Compressing objects:  21% (506/2405)           
remote: Compressing objects:  22% (530/2405)           
remote: Compressing objects:  23% (554/2405)           
remote: Compressing objects:  24% (578/2405)           
remote: Compressing objects:  25% (602/2405)           
remote: Compressing objects:  26% (626/2405)           
remote: Compressing objects:  27% (650/2405)           
remote: Compressing objects:  28% (674/2405)           
remote: Compressing objects:  29% (698/2405)           
remote: Compressing objects:  30% (722/2405)           
remote: Compressing objects:  31% (746/2405)           
remote: Compressing objects:  32% (770/2405)           
remote: Compressing objects:  33% (794/2405)           
remote: Compressing objects:  34% (818/2405)           
remote: Compressing objects:  35% (842/2405)           
remote: Compressing objects:  36% (866/2405)           
remote: Compressing objects:  37% (890/2405)           
remote: Compressing objects:  38% (914/2405)           
remote: Compressing objects:  39% (938/2405)           
remote: Compressing objects:  40% (962/2405)           
remote: Compressing objects:  41% (987/2405)           
remote: Compressing objects:  42% (1011/2405)           
remote: Compressing objects:  43% (1035/2405)           
remote: Compressing objects:  44% (1059/2405)           
remote: Compressing objects:  45% (1083/2405)           
remote: Compressing objects:  46% (1107/2405)           
remote: Compressing objects:  47% (1131/2405)           
remote: Compressing objects:  48% (1155/2405)           
remote: Compressing objects:  49% (1179/2405)           
remote: Compressing objects:  50% (1203/2405)           
remote: Compressing objects:  51% (1227/2405)           
remote: Compressing objects:  52% (1251/2405)           
remote: Compressing objects:  53% (1275/2405)           
remote: Compressing objects:  54% (1299/2405)           
remote: Compressing objects:  55% (1323/2405)           
remote: Compressing objects:  56% (1347/2405)           
remote: Compressing objects:  57% (1371/2405)           
remote: Compressing objects:  58% (1395/2405)           
remote: Compressing objects:  59% (1419/2405)           
remote: Compressing objects:  60% (1443/2405)           
remote: Compressing objects:  61% (1468/2405)           
remote: Compressing objects:  62% (1492/2405)           
remote: Compressing objects:  63% (1516/2405)           
remote: Compressing objects:  64% (1540/2405)           
remote: Compressing objects:  65% (1564/2405)           
remote: Compressing objects:  66% (1588/2405)           
remote: Compressing objects:  67% (1612/2405)           
remote: Compressing objects:  68% (1636/2405)           
remote: Compressing objects:  69% (1660/2405)           
remote: Compressing objects:  70% (1684/2405)           
remote: Compressing objects:  71% (1708/2405)           
remote: Compressing objects:  72% (1732/2405)           
remote: Compressing objects:  73% (1756/2405)           
remote: Compressing objects:  74% (1780/2405)           
remote: Compressing objects:  75% (1804/2405)           
remote: Compressing objects:  76% (1828/2405)           
remote: Compressing objects:  77% (1852/2405)           
remote: Compressing objects:  78% (1876/2405)           
remote: Compressing objects:  79% (1900/2405)           
remote: Compressing objects:  80% (1924/2405)           
remote: Compressing objects:  81% (1949/2405)           
remote: Compressing objects:  82% (1973/2405)           
remote: Compressing objects:  83% (1997/2405)           
remote: Compressing objects:  84% (2021/2405)           
remote: Compressing objects:  85% (2045/2405)           
remote: Compressing objects:  86% (2069/2405)           
remote: Compressing objects:  87% (2093/2405)           
remote: Compressing objects:  88% (2117/2405)           
remote: Compressing objects:  89% (2141/2405)           
remote: Compressing objects:  90% (2165/2405)           
remote: Compressing objects:  91% (2189/2405)           
remote: Compressing objects:  92% (2213/2405)           
remote: Compressing objects:  93% (2237/2405)           
remote: Compressing objects:  94% (2261/2405)           
remote: Compressing objects:  94% (2278/2405)           
remote: Compressing objects:  95% (2285/2405)           
remote: Compressing objects:  96% (2309/2405)           
remote: Compressing objects:  97% (2333/2405)           
remote: Compressing objects:  98% (2357/2405)           
remote: Compressing objects:  99% (2381/2405)           
remote: Compressing objects: 100% (2405/2405)           
remote: Compressing objects: 100% (2405/2405), done.        
Receiving objects:   0% (1/2506)   
Receiving objects:   1% (26/2506)   
Receiving objects:   2% (51/2506)   
Receiving objects:   3% (76/2506)   
Receiving objects:   4% (101/2506)   
Receiving objects:   5% (126/2506)   
Receiving objects:   6% (151/2506)   
Receiving objects:   7% (176/2506)   
Receiving objects:   8% (201/2506)   
Receiving objects:   9% (226/2506)   
Receiving objects:  10% (251/2506)   
Receiving objects:  11% (276/2506)   
Receiving objects:  12% (301/2506)   
Receiving objects:  13% (326/2506)   
Receiving objects:  14% (351/2506)   
Receiving objects:  15% (376/2506)   
Receiving objects:  16% (401/2506)   
Receiving objects:  17% (427/2506)   
Receiving objects:  18% (452/2506)   
Receiving objects:  19% (477/2506)   
Receiving objects:  20% (502/2506)   
Receiving objects:  21% (527/2506)   
Receiving objects:  22% (552/2506)   
Receiving objects:  23% (577/2506)   
Receiving objects:  24% (602/2506)   
Receiving objects:  25% (627/2506)   
Receiving objects:  26% (652/2506)   
Receiving objects:  27% (677/2506)   
Receiving objects:  28% (702/2506)   
Receiving objects:  29% (727/2506)   
Receiving objects:  30% (752/2506)   
Receiving objects:  31% (777/2506)   
Receiving objects:  32% (802/2506)   
Receiving objects:  33% (827/2506)   
Receiving objects:  34% (853/2506)   
Receiving objects:  35% (878/2506)   
Receiving objects:  36% (903/2506)   
Receiving objects:  37% (928/2506)   
Receiving objects:  38% (953/2506)   
Receiving objects:  39% (978/2506)   
Receiving objects:  40% (1003/2506)   
Receiving objects:  41% (1028/2506)   
Receiving objects:  42% (1053/2506)   
Receiving objects:  43% (1078/2506)   
Receiving objects:  44% (1103/2506)   
Receiving objects:  45% (1128/2506)   
Receiving objects:  46% (1153/2506)   
Receiving objects:  47% (1178/2506)   
Receiving objects:  48% (1203/2506)   
Receiving objects:  49% (1228/2506)   
Receiving objects:  50% (1253/2506)   
Receiving objects:  51% (1279/2506)   
Receiving objects:  52% (1304/2506)   
Receiving objects:  53% (1329/2506)   
Receiving objects:  54% (1354/2506)   
Receiving objects:  55% (1379/2506)   
Receiving objects:  56% (1404/2506)   
Receiving objects:  57% (1429/2506)   
Receiving objects:  58% (1454/2506)   
Receiving objects:  59% (1479/2506)   
Receiving objects:  60% (1504/2506)   
Receiving objects:  61% (1529/2506)   
Receiving objects:  62% (1554/2506)   
Receiving objects:  63% (1579/2506)   
Receiving objects:  64% (1604/2506)   
Receiving objects:  65% (1629/2506)   
Receiving objects:  66% (1654/2506)   
Receiving objects:  67% (1680/2506)   
Receiving objects:  68% (1705/2506)   
Receiving objects:  69% (1730/2506)   
Receiving objects:  70% (1755/2506)   
Receiving objects:  71% (1780/2506)   
Receiving objects:  72% (1805/2506)   
Receiving objects:  73% (1830/2506)   
Receiving objects:  74% (1855/2506)   
Receiving objects:  75% (1880/2506), 188.00 KiB | 367.00 KiB/s   
Receiving objects:  76% (1905/2506), 188.00 KiB | 367.00 KiB/s   
Receiving objects:  77% (1930/2506), 188.00 KiB | 367.00 KiB/s   
Receiving objects:  78% (1955/2506), 188.00 KiB | 367.00 KiB/s   
Receiving objects:  78% (1958/2506), 412.00 KiB | 406.00 KiB/s   
Receiving objects:  79% (1980/2506), 412.00 KiB | 406.00 KiB/s   
Receiving objects:  80% (2005/2506), 652.00 KiB | 429.00 KiB/s   
Receiving objects:  81% (2030/2506), 652.00 KiB | 429.00 KiB/s   
Receiving objects:  82% (2055/2506), 652.00 KiB | 429.00 KiB/s   
Receiving objects:  83% (2080/2506), 652.00 KiB | 429.00 KiB/s   
Receiving objects:  84% (2106/2506), 652.00 KiB | 429.00 KiB/s   
Receiving objects:  85% (2131/2506), 652.00 KiB | 429.00 KiB/s   
Receiving objects:  86% (2156/2506), 652.00 KiB | 429.00 KiB/s   
Receiving objects:  86% (2170/2506), 964.00 KiB | 468.00 KiB/s   
Receiving objects:  87% (2181/2506), 964.00 KiB | 468.00 KiB/s   
Receiving objects:  88% (2206/2506), 964.00 KiB | 468.00 KiB/s   
Receiving objects:  89% (2231/2506), 964.00 KiB | 468.00 KiB/s   
Receiving objects:  90% (2256/2506), 964.00 KiB | 468.00 KiB/s   
Receiving objects:  90% (2266/2506), 1.32 MiB | 416.00 KiB/s   
Receiving objects:  91% (2281/2506), 1.32 MiB | 416.00 KiB/s   
Receiving objects:  92% (2306/2506), 1.32 MiB | 416.00 KiB/s   
Receiving objects:  92% (2312/2506), 1.54 MiB | 411.00 KiB/s   
Receiving objects:  92% (2319/2506), 1.71 MiB | 347.00 KiB/s   
Receiving objects:  93% (2331/2506), 1.77 MiB | 306.00 KiB/s   
Receiving objects:  93% (2333/2506), 1.77 MiB | 306.00 KiB/s   
Receiving objects:  93% (2341/2506), 1.82 MiB | 222.00 KiB/s   
Receiving objects:  93% (2344/2506), 1.88 MiB | 112.00 KiB/s   
Receiving objects:  93% (2345/2506), 1.90 MiB | 97.00 KiB/s   
Receiving objects:  93% (2348/2506), 1.93 MiB | 65.00 KiB/s   
Receiving objects:  94% (2356/2506), 1.93 MiB | 65.00 KiB/s   
Receiving objects:  94% (2371/2506), 1.98 MiB | 46.00 KiB/s   
Receiving objects:  94% (2371/2506), 2.01 MiB | 34.00 KiB/s   
Receiving objects:  94% (2372/2506), 2.01 MiB | 34.00 KiB/s   
Receiving objects:  94% (2372/2506), 2.04 MiB | 29.00 KiB/s   
Receiving objects:  94% (2373/2506), 2.05 MiB | 28.00 KiB/s   
Receiving objects:  94% (2375/2506), 2.07 MiB | 22.00 KiB/s   
Receiving objects:  94% (2376/2506), 2.09 MiB | 21.00 KiB/s   
Receiving objects:  95% (2381/2506), 2.11 MiB | 19.00 KiB/s   
error: index-pack died of signal 15
fatal: index-pack failed

Looks like you are getting disconnected from the server. Is there any loss when you ping to that server? Also, you can add GIT_TRACE=1 just before the clone command, it will tell you where exactly it is breaking...! `GIT_TRACE=1 git clone ...` — Harish Talanki, Apr 19 '16 at 23:45
Therefore, when you directly use the git clone command to clone, the server memory is often full, and the clone cannot be down. [enter image description here](https://i.stack.imgur.com/Cxx2I.png) — Jsluowen, Mar 07 '20 at 03:34
If this fails via Jenkins.. How can I set GIT TRACE there? it uses the git plugin... — hilaf, Jul 18 '21 at 14:27

score 22 · Answer 1 · answered Apr 21 '16 at 16:14

I suggest you set the git clone timeout value upto a logger value, the default is 10 minutes. It would cause error if the git repo is big enough or the networking is not good.

Here is how to change the default git clone timeout value:

Source Code Management --> Git --> Add--> Additional Behaviours --> 
Advanced clone behaviors --> Timeout (in minutes) for clone and fetch operations

You can set it to 60 mins for example.

score 5 · Answer 2 · answered Jul 02 '19 at 05:09

5

Under Git select "Add" and "Select Advanced checkout behaviors"

Now specify the higher time-out period like 60 or 90 minutes (instead of default 10 minutes)

answered Jul 02 '19 at 05:09

Hemant

4,537
8
41
43

score 2 · Accepted Answer · answered Apr 21 '16 at 17:24

2

Harish Talanki's suggestion helped to narrow down the issue. It was a DNS issue on the stash. The GIT TRACE identified the same.

answered Apr 21 '16 at 17:24

Jeel

2,227
5
23
38

git clone gets hung and gets timedout

3 Answers3