1

What I want to do:

To create reports from gitlab artifacts, I want to look at all recent jobs (the past week or so) with a certain name (lighthouse in my example) and extract all artifacts.

The problem:

I got to a point where I can extract artifacts from a specific job using the python-gitlab package, when I know the jobs ID. What I didn't manage to do so far is scan my project for all jobs in the last week with the name "lighthouse" and get all the necessary jobs this way.

The question:

Is there a way to solve my problem using the gitlab-python package? I've also tried using curl as well, but as far as I can see my problem isn't solvable with curl either. Any tips? :)

sytech
  • 29,298
  • 3
  • 45
  • 86
taube5
  • 11
  • 2

1 Answers1

1

This is possible using the project pipelines list API and list pipeline jobs API.

With the python-gitlab library, it might look something like this:

import datetime
from typing import Sequence
import gitlab
from gitlab.v4.objects.pipelines import ProjectPipelineJob

gl = gitlab.Gitlab('https://gitlab.com', private_token='Your API token')

def scan_for_jobs(project_id, job_name, date_threshold) -> Sequence[ProjectPipelineJob]:
    """
    Given a project id, job name and a datetime threshold,
    returns a sequence of pipeline jobs matching the name
    where the pipeline ran at or after the threshold date.
    """
    jobs = []
    project = gl.projects.get(project_id)
    for pipeline in project.pipelines.list(as_list=False):
        created_at = datetime.datetime.fromisoformat(pipeline.created_at.replace('Z', '+00:00'))
        if created_at < date_threshold:
            break
        for job in pipeline.jobs.list(as_list=False):
            if job.name == job_name:
                jobs.append(job)
    return jobs

Suppose I wanted to get all jobs named test from the project spyoungtech/testproject from pipelines created in the last 30 days.

now = datetime.datetime.now().astimezone(datetime.timezone.utc)
threshold = now - datetime.timedelta(days=30)
test_jobs = scan_for_jobs('spyoungtech/testproject', 'test', threshold)
for job in test_jobs:
   print(job.id, job.created_at)

2196256156 2022-03-12T22:31:55.342Z
2184928033 2022-03-10T00:13:35.108Z
2184923701 2022-03-10T00:11:20.513Z
2184922621 2022-03-10T00:10:55.615Z

In your case, you might do:

now = datetime.datetime.now().astimezone(datetime.timezone.utc)
one_week_ago = now - datetime.timedelta(days=7)
lighthouse_jobs = scan_for_jobs(project_id=1234, 
                                job_name='lighthouse', 
                                date_threshold=one_week_ago
)
for job in lighthouse_jobs:
    download_artifacts(job)  # you implement this
sytech
  • 29,298
  • 3
  • 45
  • 86