1

I've got a Dash Enterprise app that uses some monthly data reports. Every month I manually upload the new data files which are stored at a Rest endpoint. I want to automate this process using GitLab. Can the GitLab CI/CD Pipeline retrieve data from a REST endpoint and upload it to the GitLab repo?

I was thinking something like this:

  1. YAML file specifies a stage in the pipeline that runs a python script
  2. The python script pulls the data from the REST endpoint example: data = requests.post('<some rest endpoint url>', params=dict(criteria='foo')).json()
  3. Somehow python or the YAML file pushes this data to the GitLab repo.

Is this possible or am I using the wrong tool for the job?

I've created a GitLab repo and created a CI/CD pipeline. I create a job in the YAML file that calls a python file which can pull data from the REST endpoint. I just don't know how this data can be pushed to the GitLab repo that the YAML and python files reside in.

russhoppa
  • 56
  • 7
  • This might be possible, but suspect then you'd have to have a RSA key which you create on the docker container which runs the pipeline so you can push without human interaction – Matthias Jan 12 '23 at 08:45
  • So it's not typical to use Gitlab pipelines to pull data into the repo from another source? Maybe I'm trying to use the wrong tool for the job then. – russhoppa Jan 12 '23 at 15:13
  • I think usually people push code to a repo, not automation, but I could be wrong. Where I work the pipeline runs tests/ sets up to environments etc. after (and in reaction to) code is pushed. For your use case seems totally prudent to just write a script which does this and run it periodically with a scheduler (airflow for example) – Matthias Jan 12 '23 at 16:10
  • "What is a CI/CD pipeline? A CI/CD pipeline automates your software delivery process. The pipeline builds code, runs tests (CI), and safely deploys a new version of the application (CD). Automated pipelines remove manual errors, provide standardized feedback loops to developers, and enable fast product iterations." https://semaphoreci.com/blog/cicd-pipeline – Matthias Jan 12 '23 at 16:10
  • Yeah I see that CI/CD pipelines are triggered by code pushes or they run on a schedule and run tests and deploy the code, but I thought maybe a job could be created in the pipeline that could pull data from an external source and save it to the repo and then continue through the test/deployment stages. Sounds like this type of job may not be possible in Gitlab. – russhoppa Jan 12 '23 at 16:30

0 Answers0