2

So I'm fairly new to airflow and have only really been using github as a fairly basic push/ pull tool rather than getting under the hood and using it for anything more complex.

That being said, now is the time I wish to do something more complex with airflow/ github.

My organisation uses Google cloud for pretty much everything and I currently use magnus to trigger my scheduled queries. For many reasons, I'm aiming to move over to airflow to perform these tasks however what I'm actually trying to do is host my source code in github and use gitpython to find the .sql files for airflow to then trigger my refresh.

I'm seemingly having trouble understanding how I can possibly 'host' my github repo in an airflow instance and then isolate a file to push to a dag task.

So, problem 1 - each time I try and connect to my remote repo, I receive a windows error

Cmd('git') not found due to: FileNotFoundError('[WinError 2] The system cannot find the file specified')
cmdline: git pull Remote_server_Address.git

I'm trying various commands but not really finding the documentation useful. As I'm aiming to host the repo in airflow (preferably within just a python instance) I'm hoping I don't need to provide a local path - but even when I try to do so, I still get the same error.

All help appreciated and apologies if it's vague.

Any other integration suggestions would also be recommended.

Thanks

Tristan
  • 49
  • 7

1 Answers1

0

It is a little hard to understand the setup you describe.

For example

isolate a file to push to a dag task

Does this mean you want a task read a specific file when you run an instance of it?

If that is the case you probably want to pass the file location (likely hosted in GCS) to the dag. This explains how.

However, a more common pattern is for something like a daily job to automatically select the file or run a query based on the date.

You could also setup a sensor that will trigger a dag when a file is added to a specific GCS folder using the gcs sensor.

Dharman
  • 30,962
  • 25
  • 85
  • 135
RayB
  • 2,096
  • 3
  • 24
  • 42