Check out git-lfs files on VSTF build

Question

I've a repository hosted on VSTS, containig a file stored through git-lfs. If I just let VSTS build checkout the repository it just downloads the git-lfs metadata file containing the ID of the file.

Here's the output how VSTS gets its source:

Syncing repository: MyRepo (Git)
Checking out c84ef2f2bbad4fa3dc70dbd4100534390b9c8f18 to d:\work\73\s
Checked out branch refs/heads/develop for repository MyRepo at commit c84ef2f2bbad4fa3dc70dbd4100534390b9c8f18

What do I need to do to checkout the real file?

Edit: I assume I need to manually call git lfs fetch after VSTS checked out the source. But how can I handle authentication (which is required by VSTS) in this case?

I agree with you that you need to call "git lfs fetch". For the Authentication part, you can create a "personal access token" for your Visual Studio Team Services account. Check this link for the details: https://msdn.microsoft.com/en-us/Library/vs/alm/Code/git/command-prompt#GetsetuptousethecommandprompttoolsCreateapersonalaccesstokenforyourVisualStudioTeamServicesaccount — Vicky - MSFT, Dec 09 '15 at 09:59
But in this case I need to define a separate service account which can be used to access the code and store its credentials on the build agent. Generally the build agent has a way to checkout the source code without the requirement of a service account. Is there a way to use the same approach from a cmd or ps script? — Pascal Berger, Dec 09 '15 at 11:37

score 7 · Accepted Answer · answered Mar 25 '17 at 23:15

7

The process has been updated yet again (March 2017). This time you need to edit the "Get Sources" part of your build definition. Enable the "Advanced Settings" option on the top right and check the option to "Checkout files from LFS".

answered Mar 25 '17 at 23:15

BrutalDev

6,181
6
58
72

mabead · Answer 2 · 2017-02-28T13:31:17.810

Update

VSTS now supports git LFS out of the box. It's just a matter of activating the option Repository / Checkout files from LFS in the build definition. It is much simpler than the solution below.

I tried Pascal's Enable Git Remote Access build task but I was not able to make it work. Calling git-lfs.exe does not crash but it does not convert the LFS files to the real files.

Here is how I was able to make it work. I first had to enable the Allow Scripts to Access OAuth Token option in my build definition. I then created a PowerShell script that pulls the LFS dependencies:

# Inspired from here: http://ss64.com/ps/syntax-set-eol.html
function Set-UnixLineEndings([string]$file)
{
    # Replace CR+LF with LF
    $text = [IO.File]::ReadAllText($file) -replace "`r`n", "`n"
    [IO.File]::WriteAllText($file, $text)

    # Replace CR with LF
    $text = [IO.File]::ReadAllText($file) -replace "`r", "`n"
    [IO.File]::WriteAllText($file, $text)
}

if ((Test-Path env:SYSTEM_ACCESSTOKEN) -eq $false)
{
    throw "OAuth token not available. Make sure that you select the option 'Allow Scripts to Access OAuth Token' in build 'Options' pane."
}

# git lfs needs the credentials of the git repository. When running
# under VSTS, these credentials are transfered to the git-lfs.exe
# application using the oauth token provided by VSTS. These
# credentials are stored in a file so that git lfs can find them.

$pwPath = Join-Path $PSScriptRoot pw.txt
$gitPwPath = $pwPath.Replace('\', '/')    # Needs to be in unix format.

$repoUri = New-Object Uri $env:BUILD_REPOSITORY_URI

git config credential.helper "store --file=$gitPwPath"
@"
https://OAuth:$env:SYSTEM_ACCESSTOKEN@$($repoUri.Host)
"@ | Set-Content $pwPath

# Again, needs to be in unix format... sigh...
Set-UnixLineEndings -file $pwPath

& ".\git-lfs.exe" pull
if ($LASTEXITCODE -ne 0)
{
    throw 'Failed to pull LFS files.'
}

This obviously assumes that you have stored git-lfs.exe in your git repository AND that this file is not tracked by LFS.

There is an [open issue](https://github.com/github/git-lfs/issues/906) with git-lfs that it doesn't support credentials passed in the remote Url. Storing token in a file like you did is the only way supported currently. — Pascal Berger, Mar 27 '16 at 08:19

Filimindji · Answer 3 · 2018-01-30T10:42:15.003

3

Update

I confirm the process changed, please ignore the following answer.

I have to say I just find that :

In your build definition, select Repository tab, and check the option Checkout files from LFS

It can't be easier.

edited Jan 30 '18 at 10:42

answered Oct 23 '16 at 15:34

Filimindji

1,453
1
16
23

Yes, in the meantime this feature is implemented out of the box in VSTS. Therefore accepting this answer as the solutions. – Pascal Berger Oct 26 '16 at 15:15

score 2 · Answer 4 · answered Apr 12 '18 at 17:56

For TFS Build 2015 Update 4 (the self-hosted flavor of this thing, as opposed to VSTS, the cloud service), the "Checkout files from LFS" option does not exist, and as such, we had to do this:

Check Allow Scripts to Access OAuth Token on the Options tab.

Commit this batch file into the repository somewhere:

REM This script is intended to fetch large (LFS) files during the TFS Build process.

REM Solution derived from https://github.com/Microsoft/vsts-agent/issues/1134

git config --unset-all http.extraheader
git config --add http.extraheader "AUTHORIZATION: bearer %SYSTEM_ACCESSTOKEN%"
git lfs fetch
git lfs checkout
git config --unset http.extraheader

Add a build step of a Batch Script, and run the above script.

Thanks for the `extraheader` syntax. +1 – InteXX Jun 28 '18 at 01:56 — InteXX, Jun 28 '18 at 01:56

Pascal Berger · Answer 5 · 2015-12-20T12:34:02.713

0

VSTS has now an Allow Scripts to Access OAuth Token option. With this option set on the build definition the OAuth is available to build scripts.

I've created an extension containing build tasks which change the remote Url to use the OAuth token to access the remote repository.

edited Dec 20 '15 at 12:34

answered Dec 11 '15 at 07:47

Pascal Berger

4,262
2
30
54

Check out git-lfs files on VSTF build

5 Answers5

Linked