How do I fix git fsync errors when checking out code on mapped network drives

Question

I have a Jenkins job that checks out git code to a mapped network file share so that multiple executors can work on and test code in parallel. I would estimate that a good 90% of the time everything is wonderful and works and then seemingly out of nowhere I will come across the following.

ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command ""C:\Program Files\Git\bin\git.exe" fetch --tags --progress https://bitbucket.org/somecompany/some.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: remote: Counting objects: 104850, done.        
remote: Compressing objects:   0% (1/24671)           
remote: Compressing objects:   1% (247/24671)           
remote: Compressing objects:   2% (494/24671)           
remote: Compressing objects:   3% (741/24671)           
...        
remote: Compressing objects:  97% (23931/24671)           
remote: Compressing objects:  98% (24178/24671)           
remote: Compressing objects:  99% (24425/24671)           
remote: Compressing objects: 100% (24671/24671)           
remote: Compressing objects: 100% (24671/24671), done.        
Receiving objects:   0% (1/104850)   
Receiving objects:   1% (1049/104850), 244.01 KiB | 464.00 KiB/s   
Receiving objects:   1% (1427/104850), 244.01 KiB | 464.00 KiB/s   
Receiving objects:   2% (2097/104850), 440.01 KiB | 285.00 KiB/s   
Receiving objects:   2% (2456/104850), 440.01 KiB | 285.00 KiB/s   
Receiving objects:   3% (3146/104850), 648.01 KiB | 252.00 KiB/s   
Receiving objects:   3% (3483/104850), 648.01 KiB | 252.00 KiB/s   
...
Receiving objects:  97% (101705/104850), 11.32 MiB | 1.78 MiB/s   
Receiving objects:  98% (102753/104850), 11.32 MiB | 1.78 MiB/s   
Receiving objects:  98% (103514/104850), 11.96 MiB | 1.91 MiB/s   
Receiving objects:  99% (103802/104850), 12.05 MiB | 1.82 MiB/s   
Receiving objects:  99% (104664/104850), 12.05 MiB | 1.82 MiB/s   
remote: Total 104850 (delta 78967), reused 104782 (delta 78936)        
Receiving objects: 100% (104850/104850), 12.05 MiB | 1.82 MiB/s   
Receiving objects: 100% (104850/104850), 12.40 MiB | 1.05 MiB/s, done.
Resolving deltas:   0% (0/78967)   
Resolving deltas:   1% (804/78967)   
Resolving deltas:   1% (1193/78967)   
Resolving deltas:   2% (1581/78967)   
Resolving deltas:   2% (2191/78967)   
Resolving deltas:   3% (2428/78967)   
Resolving deltas:   3% (3006/78967)   
...
Resolving deltas:  97% (76616/78967)   
Resolving deltas:  97% (77322/78967)   
Resolving deltas:  98% (77400/78967)   
Resolving deltas:  99% (78188/78967)   
Resolving deltas:  99% (78328/78967)   
Resolving deltas: 100% (78967/78967)   
Resolving deltas: 100% (78967/78967), done.
fatal: fsync error on '.git/objects/pack/tmp_idx_nU1ucb': Bad file descriptor
fatal: index-pack failed

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2002)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1721)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:405)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:614)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
    at hudson.remoting.UserRequest.perform(UserRequest.java:210)
    at hudson.remoting.UserRequest.perform(UserRequest.java:53)
    at hudson.remoting.Request$2.run(Request.java:358)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at hudson.remoting.Engine$1$1.run(Engine.java:94)
    at java.lang.Thread.run(Unknown Source)
    Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from mcusdev31.mainman.dcs/172.21.10.11:50789
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
        at hudson.remoting.UserResponse.retrieve(UserRequest.java:389)
        at hudson.remoting.Channel.call(Channel.java:955)
        at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
        at sun.reflect.GeneratedMethodAccessor868.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
        at com.sun.proxy.$Proxy120.execute(Unknown Source)
        at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1146)
        at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
        at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
        at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
        at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
        at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
        at hudson.security.ACL.impersonate(ACL.java:290)
        at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        ... 1 more

I have no idea what suddenly causes this or where it comes from. Here it is stated that:

The proper solution is not to store your Git repositories on network drives. (I'm not trying to be snarky. This is explicitly not a supported scenario.)

But is this really true? Is there no way around this? This seems to work reasonably well aside from the occasional hiccup, but I'm hoping to produce a more deterministic build system that is capable of running code testing tasks in parallel without having to constantly checkout or move our entire repository. Does anyone know what my problem might be or have they found a more consistent way of doing this?

Yes, it's really true: Git depends on certain file system semantics that most remote-access file systems do not support. If nominally-shared locations are actually private (no two users at the same time) and some other poorly enumerated conditions are all met, it *can* work reliably. If not—and since the conditions are difficult to determine, they tend to fall into "not" at times—you get the behavior you are observing. (In this case it means some other user of the shared area did something to the underlying file while Git thought that only Git would be doing things.) — torek, Jun 29 '18 at 06:31
Does this happen even if no one else is accessing the same folder in the network file system? — Slushba132, Jun 29 '18 at 20:12
It generally shouldn't. I'm making some rather broad assumptions about the underlying file system implementation here though, e.g., that it's keeping in-memory information for all the magic. If it doesn't do that (if it discards data structures due to memory pressure for instance, keeping things in-core only as a cache) you could get this kind of behavior if the server is stressed. — torek, Jun 29 '18 at 21:01

How do I fix git fsync errors when checking out code on mapped network drives

0 Answers0