2

I'm trying to get a newly configured VSTS private build agent download files stored in Git LFS, but it only seems to download the metadata for each file, with file contents ending up like this:

version https://git-lfs.github.com/spec/v1 
oid sha256:1c4fa4fba8880241858d324b17b6b95512b65f98ea3ea1e959659f0176e45ec2 
size 23552

The VSTS build agent is of version 2.123.0.

In order to get Git LFS working at all, keeping in mind that this is a private agent running on an internal server, I had to manually install it, and it has version 2.3.3.

The following commands are being run by the agent;

Initialization:

2017-10-11T13:19:25.8153872Z ##[command]git version
2017-10-11T13:19:25.9089878Z git version 2.12.2.windows.3
2017-10-11T13:19:25.9245879Z ##[command]git lfs version
2017-10-11T13:19:26.1749886Z git-lfs/2.3.3 (GitHub; windows amd64; go 1.8.3; git c9d3beb0)
2017-10-11T13:19:26.1749886Z ##[command]git config --get remote.origin.url
2017-10-11T13:19:26.2529886Z ##[command]git config gc.auto 0
2017-10-11T13:19:26.2997886Z ##[command]git config --get-all http.https://aaa.visualstudio.com/bbb/_git/ccc.extraheader
2017-10-11T13:19:26.3621886Z ##[command]git config --get-all http.proxy
2017-10-11T13:19:26.4089886Z ##[command]git lfs install --local
2017-10-11T13:19:27.2705886Z Updated git hooks.
2017-10-11T13:19:27.2705886Z Git LFS initialized.

Main Git fetch command:

2017-10-11T13:19:27.3027886Z ##[command]git -c http.extraheader="AUTHORIZATION: bearer ********" fetch --tags --prune --progress --no-recurse-submodules origin
[...]
2017-10-11T13:19:29.7987886Z Receiving objects:   0% (1/149024)

Git LFS fetch command:

2017-10-11T13:21:24.5211886Z ##[command]git -c http.extraheader="AUTHORIZATION: bearer ********" lfs fetch origin 3893c0f7d8847adc21182dfd6d10c5d065bda565
2017-10-11T13:21:24.7745886Z Fetching 3893c0f7d8847adc21182dfd6d10c5d065bda565
2017-10-11T13:21:25.9157886Z ##[command]git checkout --progress --force 3893c0f7d8847adc21182dfd6d10c5d065bda565
2017-10-11T13:21:27.1799886Z Checking out files:   4% (550/11027)

Final submodule (where the LFS files actually reside) sync:

2017-10-11T13:21:47.8469886Z ##[command]git submodule sync
2017-10-11T13:21:49.2977886Z ##[command]git -c http.https://aaa.visualstudio.com.extraheader="AUTHORIZATION: bearer ********" submodule update --init --force
2017-10-11T13:21:49.9997886Z Submodule 'ExternalSoftware' (https://aaa.visualstudio.com/bbb/_git/ccc.ExternalSoftware) registered for path 'ExternalSoftware'
2017-10-11T13:21:51.5285886Z Cloning into 'E:/agent/_work/1/s/ExternalSoftware'...
2017-10-11T13:21:53.3537886Z Submodule path 'ExternalSoftware': checked out '71ce8e95c4f43447c1d1419be7ffb7829a517c2f'
2017-10-11T13:21:53.3693886Z ##[section]Finishing: Get Sources

So, what am I doing wrong to only get the metadata for the LFS-based files, and not the actual file content?

Seems like the issue I'm having is slightly related to Check out git-lfs files on VSTF build, but still, that problem was solved by the same out-of-the-box Git LFS support that I'm using.

UPDATE:
As a test, I tried a VSTS agent on a different server, and that was able to download the files.
That is, I don't have login rights to the server, so I haven't verified the file content, but the following was logged:

2017-10-12T07:44:26.4608720Z ##[command]git -c http.extraheader="AUTHORIZATION: bearer ********" lfs pull
2017-10-12T07:44:29.2051354Z 
2017-10-12T07:44:29.4005140Z Git LFS: (0 of 201 files) 0 B / 36.67 MB                                       
2017-10-12T07:44:29.6005312Z Git LFS: (0 of 201 files) 0 B / 36.67 MB                                       
2017-10-12T07:44:29.8021911Z Git LFS: (0 of 201 files) 0 B / 36.67 MB                                       
2017-10-12T07:44:30.0024064Z Git LFS: (0 of 201 files) 256.00 KB / 36.67 MB                                 
2017-10-12T07:44:30.2025273Z Git LFS: (0 of 201 files) 960.00 KB / 36.67 MB                                 
2017-10-12T07:44:30.4029958Z Git LFS: (0 of 201 files) 1.38 MB / 36.67 MB                                   
2017-10-12T07:44:30.6039364Z Git LFS: (0 of 201 files) 1.62 MB / 36.67 MB                                   
2017-10-12T07:44:30.8049023Z Git LFS: (1 of 201 files) 1.92 MB / 36.67 MB                                   
[...]
2017-10-12T07:45:10.4673396Z Git LFS: (275 of 275 files) 103.98 MB / 103.98 MB                              
2017-10-12T07:45:10.4943356Z ##[section]Finishing: Pull external software LFS files

For other third-party "capability" reasons, I can't yet use that agent, though, but it proves that the mechanism should be able to work for my repositories.

UPDATE:
The full log was requested by @Marina-MSFT, so here goes (it's from a recent run, so timestamps are a bit different from above): 

2017-10-13T10:26:07.0921581Z ##[section]Starting: Get Sources
2017-10-13T10:26:07.1389599Z Syncing repository: ccc (TfsGit)
2017-10-13T10:26:07.1545605Z Prepending Path environment variable with directory containing 'git.exe'.
2017-10-13T10:26:07.1545605Z ##[command]git version
2017-10-13T10:26:07.2481641Z git version 2.12.2.windows.3
2017-10-13T10:26:07.2637647Z ##[command]git lfs version
2017-10-13T10:26:07.3271672Z git-lfs/2.3.3 (GitHub; windows amd64; go 1.8.3; git c9d3beb0)
2017-10-13T10:26:07.3583684Z ##[command]git config --get remote.origin.url
2017-10-13T10:26:07.4363714Z ##[command]git config gc.auto 0
2017-10-13T10:26:07.4987738Z ##[command]git config --get-all http.https://aaa.visualstudio.com/bbb/_git/ccc.extraheader
2017-10-13T10:26:07.5923774Z ##[command]git config --get-all http.proxy
2017-10-13T10:26:07.6859810Z ##[command]git lfs install --local
2017-10-13T10:26:08.2328280Z Updated git hooks.
2017-10-13T10:26:08.2338281Z Git LFS initialized.
2017-10-13T10:26:08.2650293Z ##[command]git -c http.extraheader="AUTHORIZATION: bearer ********" fetch --tags --prune --progress --no-recurse-submodules origin
2017-10-13T10:26:08.9358551Z ##[command]git -c http.extraheader="AUTHORIZATION: bearer ********" lfs fetch origin 440cb61e73a938c4c83da56823b53d13198ca9b9
2017-10-13T10:26:09.2744832Z Fetching 440cb61e73a938c4c83da56823b53d13198ca9b9
2017-10-13T10:26:11.4949033Z ##[command]git checkout --progress --force 440cb61e73a938c4c83da56823b53d13198ca9b9
2017-10-13T10:26:11.8381165Z HEAD is now at 440cb61e7... [some commit]
2017-10-13T10:26:12.8444104Z ##[command]git submodule sync --recursive
2017-10-13T10:26:14.2328638Z Synchronizing submodule url for 'ExternalSoftware'
2017-10-13T10:26:14.8100860Z ##[command]git -c http.https://aaa.visualstudio.com.extraheader="AUTHORIZATION: bearer ********" submodule update --init --force --recursive
2017-10-13T10:26:17.0409718Z Submodule path 'ExternalSoftware': checked out '71ce8e95c4f43447c1d1419be7ffb7829a517c2f'
2017-10-13T10:26:17.3529838Z ##[command]git config http.https://aaa.visualstudio.com/bbb/_git/ccc.extraheader "AUTHORIZATION: bearer ********"
2017-10-13T10:26:17.3997856Z ##[command]git config http.https://aaa.visualstudio.com/bbb/_git/ccc.extraheader "AUTHORIZATION: bearer ********"
2017-10-13T10:26:17.4621880Z ##[section]Finishing: Get Sources

2017-10-13T10:26:17.4621880Z ##[section]Starting: Pull external software LFS files
2017-10-13T10:26:17.4933892Z ==============================================================================
2017-10-13T10:26:17.4933892Z Task         : Command Line
2017-10-13T10:26:17.4933892Z Description  : Run a command line with arguments
2017-10-13T10:26:17.4933892Z Version      : 1.1.3
2017-10-13T10:26:17.4933892Z Author       : Microsoft Corporation
2017-10-13T10:26:17.4933892Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkID=613735)
2017-10-13T10:26:17.4933892Z ==============================================================================
2017-10-13T10:26:17.5713922Z ##[command]git -c http.extraheader="AUTHORIZATION: bearer ********" lfs pull
2017-10-13T10:26:18.3030596Z 
2017-10-13T10:26:18.4840777Z Git LFS: (0 of 275 files) 0 B / 103.98 MB                                      
2017-10-13T10:26:18.6790972Z Git LFS: (0 of 275 files) 0 B / 103.98 MB                                      
[...]
2017-10-13T10:26:33.2805572Z Git LFS: (0 of 275 files) 0 B / 103.98 MB                                      
2017-10-13T10:26:33.3877660Z ##[error]^C

2017-10-13T10:26:33.3877660Z Git LFS: (0 of 275 files) 0 B / 103.98 MB                                      
2017-10-13T10:26:33.3877660Z ##[error]The operation was canceled.
2017-10-13T10:26:33.4033666Z ##[section]Finishing: Pull external software LFS files
Morten Grøtan
  • 103
  • 2
  • 10
  • Does the two commits `1c4fa4f` and `71ce8e9` both for the submodule repo ExternalSoftware? And which commit is newer? If you didn't update the submodule in the main repo, please update. And can you show the complete log for Get sources step? – Marina Liu Oct 12 '17 at 09:06
  • @Marina-MSFT, the "1c4fa4f" is not a commit, AFAIK, but the start of the sha256 of one of the LFS files. They are all different. The "71ce8e9" you mention, though, is the commit of the updated submodule in the main repo. The agent has already updated the submodule, as per the commands shown in the full log now included in the original post. – Morten Grøtan Oct 13 '17 at 11:32
  • I can check submodule and it's lfs files successful in Get sources step. First please check if the submodule repo has lfs files by cloning the repo locally and use `git lfs ls-files` to check. Then please make sure **Checkout submodules** and **Checkout files from LFS** options are selected. My detail logs is here https://1drv.ms/t/s!ApIKkY1MTjMRgmn8Kj6b7ZrFBxlh. – Marina Liu Oct 17 '17 at 07:20
  • @MortenGrøtan Did you manage to resolve this? I have the same problem running on an on-premise TFS build agent. I get LFS files on the parent repo but not inside submodules. – Per Salmi Mar 14 '18 at 16:01

1 Answers1

0

Basically, it's cloning the submodule but bypassing LFS for the submodule. There's been a lot of discussion on GitHub about how it should work over the years.

Try adding a Command Line task and running git lfs pull --recursive as the first step in your build.

Daniel Mann
  • 57,011
  • 13
  • 100
  • 120
  • No such luck: 2017-10-11T15:39:17.1120000Z ##[command]git lfs pull --recursive 2017-10-11T15:39:17.2066019Z 2017-10-11T15:39:17.2066019Z git lfs pull [options] [] 2017-10-11T15:39:17.2066019Z Error: unknown flag: --recursive – Morten Grøtan Oct 11 '17 at 15:42
  • If I run without the "--recursive" flag, it doesn't fail, but doesn't do anything either: 2017-10-11T15:49:06.0093960Z ##[command]git lfs pull 2017-10-11T15:49:08.5354369Z ##[section]Finishing: Fetch Git LFS properly – Morten Grøtan Oct 11 '17 at 15:56