How to specify depth in checkout(scm) in scripted pipeline

Question

our project repo is very big (2.5GB). Hence upon checkout(scm) step in the scripted pipeline, the code takes longer time to clone from GIT. and We are facing the below errors, due to GIT trying to fetch the entire history.

I have tried so far with checkout(scm), I read in https://jenkins.io/doc/pipeline/steps/workflow-scm-step/ that there is an option called depth, by which we can download only the recent commits.

But i don't know the syntax for it.

node(nodeName) {
    try {
    deleteDir()
    checkout(scm)
        ....
        ....
    }
    catch(Exception ex) {
        throw ex
    }    
}

If the cloning times reduces, it would be a lot beneficial. Upon executing the line checkout(scm),

We are sometimes getting the below error.

using credential Servicejenkins_build
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository 
hudson.plugins.git.GitException: Command "C:\Git\cmd\git fetch --no-tags --progress https://gitlab.com/../../supportforpc.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: remote: Enumerating objects: 1           
remote: Enumerating objects: 24671, done.        
remote: Counting objects:   0% (1/24671)           
remote: Counting objects:   1% (247/24671)           
remote: Counting objects:   2% (494/24671)           
remote: Counting objects:   3% (741/24671)           
remote: Counting objects:   4% (987/24671)           
remote: Counting objects:   5% (1234/24671)           
remote: Counting objects:   6% (1481/24671)  
.....
....
Counting objects:   100% (24671/24671) 
remote: Compressing objects:   0% (1/10279)           
remote: Compressing objects:   1% (103/10279)           
remote: Compressing objects:   2% (206/10279)           
remote: Compressing objects:   3% (309/10279)           
remote: Compressing objects:   4% (412/10279)           
remote: Compressing objects:   5% (514/10279)           
remote: Compressing objects:   6% (617/10279)           
remote: Compressing objects:   7% (720/10279)           
remote: Compressing objects:   8% (823/10279)           
remote: Compressing objects:   9% (926/10279)           
remote: Compressing objects:  10% (1028/10279) 
....
....
remote: Compressing objects:  100% (10279/10279) 
Receiving objects:   0% (1/24671)   
Receiving objects:   1% (247/24671)   
Receiving objects:   2% (494/24671)   
Receiving objects:   3% (741/24671)   
Receiving objects:   4% (987/24671)   
Receiving objects:   5% (1234/24671)   
Receiving objects:   6% (1481/24671)
....
....
Receiving objects:   43%

    fatal: index-pack failed
    error: RPC failed; curl 56 SSL read: 
    error:00000000:lib(0):func(0):reason(0), errno 10054

Hence I thought, using depth 1 in checkout(scm) might solve the issue. But i dont know the syntax in scripted pipeline.

towel · Answer 1 · 2019-10-03T12:01:01.917

You can enable a shallow clone (no history and only the most recent commit is fetched) in the scm object before the checkout:

node(nodeName) {
    try {
    deleteDir()
    scm.extensions << [$class: 'CloneOption', shallow: true]
    checkout(scm)
        ....
        ....
    }
    catch(Exception ex) {
        throw ex
    }    
}

However, I would advise you to set up and maintain a reference repository, which is faster in an order of magnitude:

In a terminal window clone your repository into a mirror. This repository will only contain git objects:

$ git clone --mirror https://gitlab.com/../../supportforpc.git
Cloning into bare repository 'supportforpc.git'...
remote: Enumerating objects: 6578, done.
remote: Counting objects: 100% (6578/6578), done.
remote: Compressing objects: 100% (1561/1561), done.
remote: Total 739260 (delta 5791), reused 5046 (delta 5013), pack-reused 732682
Receiving objects: 100% (739260/739260), 3.49 GiB | 3.78 MiB/s, done.
Resolving deltas: 100% (562236/562236), done.

Create a new job in Jenkins to update the mirror repository periodically. Use only the fetch command when updating the mirror:
```
sh "git fetch --all --prune"
```

Tell the scm object to make use of the mirror repository as reference. The remote repository is queried for new commits after the reference is read, so you don't have to worry about always keeping the mirror up-to-date:

node(nodeName) {
    try {
    deleteDir()
    scm.extensions << [$class: 'CloneOption', reference: "<your-server>:git/<where-you-put-the-mirror-repo>"]
    checkout(scm)
        ....
        ....
    }
    catch(Exception ex) {
        throw ex
    }    
}

After that setup you'll see that your whole repository is cloned in a matter of seconds, plus you get to keep all the repository's history. You can test the clone for yourself if you manually clone from the mirror repository you created in step 1:

$ git clone <mirror-repository-directory> <some-dir>`
Cloning into '<some-dir>'...
done.
Checking out files: 100% (15055/15055), done.

A note regarding the modification of the default scm object: if you prefer not to change it you can create a new object for the checkout as shown in this answer:

checkout([
    $class: 'GitSCM',
    branches: scm.branches,
    doGenerateSubmoduleConfigurations: scm.doGenerateSubmoduleConfigurations,
    extensions: scm.extensions + [$class: 'CloneOption', reference: "<your-server>:git/<where-you-put-the-mirror-repo>"],
    userRemoteConfigs: scm.userRemoteConfigs
])

You place the `scm.extensions` line before `checkout`. The `git` commands are actually meant to be run in a terminal, except the fetch command. I'll update the answer. — towel, Oct 03 '19 at 09:13
yeah thanks. I was hoping to see an example of these commands, in a scripted pipeline. — murthi, Oct 03 '19 at 09:15
I also have a big repository :) When it's done making the mirror repository I'll paste the clone's output. — towel, Oct 03 '19 at 09:18
thanks dude, i will try this and see for a few days, then will upvote this answer. — murthi, Oct 03 '19 at 10:13
It's alright - only upvote an answer if you find it helpful. For my use case a reference repository works great, but it might not be so for others. — towel, Oct 03 '19 at 12:05
Why is a mirror repository faster than a shallow clone/checkout? Shouldn't it be just as fast? — enanone, Jul 27 '22 at 12:02
For a small code base a mirror repository isn't worth the hassle, though from my experience the git objects are deployed much faster from a mirror repository than a shallow clone. If your repository holds hundreds of megabytes or more you'll immediately feel the difference between a mirror repository and a shallow clone. I don't really know what happens behind the scenes to give a explanation so take this with a grain of salt: I guess this is because the git objects are readily available and don't have to be downloaded. — towel, Jul 27 '22 at 20:19

score 0 · Answer 2 · edited Sep 28 '21 at 08:58

0

checkout changelog: false, poll: false, scm: [$class: 'GitSCM', branches: [[name: '*/branchname'], [$class: 'CloneOption', depth: 1, noTags: true, reference: '', shallow: true]], userRemoteConfigs: [[credentialsId: 'credentialId', url: 'git@repourl']]]

credentialId can be found by hitting http://yourjenkinsinstall/credentials

edited Sep 28 '21 at 08:58

m4n0

29,823
27
76
89

answered Sep 27 '21 at 12:52

Idris Merchant

21
1

1

This results in `no known implementation of interface java.util.List is named CloneOption`. because you are passing the `CloneOption` to the `branches` argument. The documentation at https://www.jenkins.io/doc/pipeline/steps/workflow-scm-step/ states that `CloneOption` is passed to the `extensions` argument. – cowlinator Feb 18 '22 at 23:13

score 0 · Answer 3 · answered Jun 26 '23 at 15:58

According to towel's answer, you can also avoid all tags from your git repository by adding in the extension CloneOption the argument noTags, here's a look to the checkout object :

checkout([
    $class: 'GitSCM', 
    branches: [[name: '*/master']], 
    extensions: [[$class: 'CloneOption', noTags: true, shallow: true, depth: 1, timeout: 30]],
    userRemoteConfigs: [[credentialsId: 'your-credentials', url: 'your-repository']]
])

Here's a look at the documentation :

noTags : boolean Deselect this to perform a clone without tags, saving time and disk space when you just want to access what is specified by the refspec.

Link : https://www.jenkins.io/doc/pipeline/steps/params/gitscm/

How to specify depth in checkout(scm) in scripted pipeline

3 Answers3