How to list all the files present within GitLab repository subfolder using API and recurse over it to read them one by one?

Question

I am developing a project using Nodejs, as per the requirement I would like to list the files present within the subfolder of the GitLab repository using the GitLab API.

The subfolder contains many files so I would like to list all the file names and then traverse over the files one by one to read their content.

I am unable to list all the files present within a subfolder. However, I am able to read the contents of the specific file by providing the specific file name.

Following is the folder structure that I have within my GitLab repository:

projectName
  - .FirstSubFolder
    - SecondSubFolder
      - ThirdSubFolder

Following is the API that I am trying which I would like to read all files within my ThirdSubFolder:

https://{gitlabURL}/api/v4/projects/{id}/repository/files/%2EFirstSubFolder%2FSecondSubFolder%2FThirdSubFolder?ref=master&private_token={api}

This returns me:

{
    "message": "404 File Not Found"
}

However, if I try to read the specific file within ThirdSubFolder then I am able to get the contents:

https://{gitlabURL}/api/v4/projects/{id}/repository/files/%2EFirstSubFolder%2FSecondSubFolder%2FThirdSubFolder%2FmyFile.feature/raw?ref=master&private_token={api}

Can someone please assist me in how to read all the files present within the subfolder of the GitLab repository using the API?

Tolis Gerodimos · Accepted Answer · 2022-02-16T18:17:18.657

I am developing a project using Nodejs, as per the requirement I would like to list the files present within the subfolder of the GitLab repository using the GitLab API.

You could use the Gitlab Repository API https://docs.gitlab.com/ee/api/repositories.html#list-repository-tree

GET /projects/:id/repository/tree

By combining the API with curl we can get the files in a certain subdirectory

curl -s  --header "PRIVATE-TOKEN: <access_token>" "https://gitlab.com/api/v4/projects/<project_id>/repository/tree?ref=<target_branch>&path=path/to/subdirectory" | jq -r .[].path

Example output would be

path/to/file1
path/to/file2

And finally to read their contents I would suggest the following script

curl -s  --header "PRIVATE-TOKEN: <access_token>" "https://gitlab.com/api/v4/projects/<project_id>/repository/tree?ref=master&path=path/to/subdirectory" | jq -r .[].path |
    while read -r path
    do
            f_path=$(echo $path | sed 's:/:%2F:')
            curl -s  --header "PRIVATE-TOKEN: <access_token>" "https://gitlab.com/api/v4/projects/<project_id>/repository/files/$f_path/raw?ref=master"
    done

How to list all the files present within GitLab repository subfolder using API and recurse over it to read them one by one?

1 Answers1