Download files through GitLab API that has been manually uploaded

Question

We have a self-hosted GitLab server and are working on automating our builds and releases. We have many old releases that we have built before using GitLab CI. Some of these should be included in a release package for a certain software. The releases are not located on any server that is easy to access, so it would be very easy if they could be accessed from our GitLab server.

It is possible to access tags from the API and to get artifacts from the build jobs. It doesn't seem possible to add build artifacts manually, so there is no way of using this for old releases.

It is possible to upload files to the release notes of a tag. These are very simple to download through the webpage, but I can't find any way of downloading these through the API. There is this API endpoint:

https://docs.gitlab.com/ee/api/projects.html#upload-a-file

but there is no "download-a-file".

Is there an easy way to upload files to our self-hosted GitLab and then to download them through the API?

All of our repositories has visibility set to private. If you try to access a link like this one, without being logged in:

http://www.example.com/group/my-project/uploads/443568a8641b1b48fc983daea27d36c0/myfile.zip

Then you get redirected to the login page.

MrBerta · Accepted Answer · 2019-11-22T16:21:09.360

Downloading files from the GitLab API is not possible. Here is GitLab's issue to track this feature:

https://gitlab.com/gitlab-org/gitlab/issues/25838

------

As a workaround for now, I am using a python script to simply log in to the page and get the file. This requires the username and password of a user. I set up a new user that only has access to the specific project where it is needed, so it should be safe enough to use in the build script of this project.

import argparse
import sys
import mechanize

def login_and_download(url, user, password):
    br = mechanize.Browser()
    br.set_handle_robots(False)
    br.open(url)
    br.select_form(id = 'new_user')
    br['user[login]'] = user
    br['user[password]'] = password
    response = br.submit()
    filename = url.split('/')[-1] #Last part of url
    with open(filename, 'wb') as f:
        f.write(response.get_data())

def main():
    parser = argparse.ArgumentParser(description = 'Downloads files from a private git repo')
    parser.add_argument('url',  help = 'The URL to download')
    parser.add_argument('user',  help = 'The username to use')
    parser.add_argument('password',  help = 'The password to use')
    args = parser.parse_args()

    if len(sys.argv) == 1:
        parser.print_help()
        sys.exit(1)

    login_and_download(args.url, args.user, args.password)

if __name__ == '__main__':
    main()

VonC · Answer 2 · 2023-05-04T16:51:42.747

Here is another example of python script, still from issue 25838.

It translates the curl call:

curl --request GET 'http://127.0.0.1:3000/root/my-project/uploads/d3d527ac296ae1b636434f44c7f57272/IMG_3935.jpg' \
     --header 'Authorization: Bearer YOUR_API_TOKEN'

import requests
from bs4 import BeautifulSoup

with requests.Session() as http_session:

    # my login page url
    login_url = [LOGIN_URL]
    login_page = http_session.get(login_url)
    login_inputs = BeautifulSoup(login_page.content, features='html.parser').find_all('input')

    # get the authenticity token from the login page
    for login_input in login_inputs:
        if 'token' in login_input['name']:
            auth_token = login_input['value']
            break

    # send some user-agent header just in case
    headers = {"User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"}

    # see your login page source for the required data ...
    data = {
        "username": [MYUSERNAME],
        "password": [MYPASSWORD],
        "authenticity_token": auth_token,
        "remember_me": "1",
        "commit": "Sign in"
    }

    # ... as well as the login endpoint
    login_endpoint = [LOGIN_ENDPOINT_URL]

    # authenticate
    http_session.post(login_endpoint, data=data, headers=headers)

    # myfile_url is the url for my wanted file
    myfiledata = http_session.get(myfile_url).content

The same issue 25838 confirms:

For now the /uploads endpoint is unusable for automation as it's basically write-only in that context, and most sane users won't resort to web browser instrumentation to download their assets.

I suggest a workaround by publishing the asset into a generic package:

To upload a file:

curl --header "Private-Token: glpat-XXXXXXXXXXXXXXXXXXX" --upload-file path/to/file.ext https://gitlab.com/api/v4/projects/:project-id/packages/generic/:package-name/:package-version/:filename

To download it:

curl --header "Private-Token: glpat-XXXXXXXXXXXXXXXXXXX" https://gitlab.com/api/v4/projects/:project-id/packages/generic/:package-name/:package-version/:filename

Arguably easier than dealing with the /uploads endpoint and /releases. > For now it does cover my use case, which is to ship internal releases consumed by a build system.

Download files through GitLab API that has been manually uploaded

2 Answers2