0

I'm trying to get a full list of Projects and groups in out Gitlab cloud account. I'm currently using their documentation as reference (bear in mind though I'm no developer) and using Linux command line to do so. Here's the documentation I'm trying to use:

https://docs.gitlab.com/ee/api/projects.html

https://docs.gitlab.com/ee/api/groups.html#list-a-groups-projects

I'm using the following command to get the data and parse in a readable format that I will export to csv or spreadsheet afterwards:

curl --header "PRIVATE-TOKEN: $TOKEN" "https://gitlab.com/api/v4/projects/?owned=yes&per_page=1000&page=1" | python -m json.tool | grep -E "http_url_to_repo|visibility" | awk '!(NR%2){print$0p}{p=$0}' | awk '{print $4,$2}' | sed -E 's/\"|\,//g' > gitlab.txt

My problem is that the code only return about 100 of the 280 repositories we have. It doesn't seem to get it recursively from all the groups and subgroups. Any ideas on how I can improve this search to get everything ?

Thank you

furas
  • 134,197
  • 12
  • 106
  • 148
Adonist
  • 153
  • 1
  • 6
  • 1
    maybe it can get only 100 and using `per_page=1000` can't change it - and you should run it two times - first with `page=1` and next with `page=2` – furas Aug 19 '21 at 12:35
  • 1
    Does this answer your question? [how to retrive all commits of a special branch with gitlab api v4?](https://stackoverflow.com/questions/57388538/how-to-retrive-all-commits-of-a-special-branch-with-gitlab-api-v4) – phd Aug 19 '21 at 12:35
  • 1
    https://stackoverflow.com/search?q=%5Bgitlab-api%5D+paging – phd Aug 19 '21 at 12:36
  • 1
    GitLab API has a max 100 per page: https://docs.gitlab.com/ee/api/#pagination – phd Aug 19 '21 at 12:36
  • Thank you for that, seems that's a big difference between version 3 and 4 of the api. – Adonist Aug 19 '21 at 13:26

1 Answers1

1

It seems it can get only 100 per page so you will have to run it two times - first with page=1 and next with page=2. And for second page you will need >> to append to existing file gitlab.txt

curl --header "..." "https://...&per_page=100&page=1" | ...  > gitlab.txt

curl --header "..." "https://...&per_page=100&page=2" | ...  >> gitlab.txt

Or you will have to write script which first get all pages and later send it to pipe. You may also try to use for-loop in bash

furas
  • 134,197
  • 12
  • 106
  • 148
  • That did the trick all right. Seems to be a limitation in version 4 that didn't have on version 3. Thank you for the help! – Adonist Aug 19 '21 at 13:27