I need to upload a log file through the airflow web server UI and parse that log file in a DAG.
My requirement is a button on UI, which upon clicking will open a file selector to upload a file to airflow. then use this file in the DAG.
Similarly, we use the Trigger DAG w/ config option.
Is this Possible?
Asked
Active
Viewed 264 times
-1
Prinz Piuz
- 69
- 9
2 Answers
0
you may try it using the "Admin -> Connections" menu in the Airflow Webserver UI to create a connection to a file storage service, such as Amazon S3 or Google Cloud Storage. Once the connection is set up, you can use the "PythonOperator" or "BashOperator" in your DAG to download the data file from the storage service to the local file system where the DAG is running.
Another option is to use the Airflow Webserver UI's "Admin -> Variables" menu to define a variable that contains the path to the data file. Then, in your DAG, you can use the "Variable" operator to retrieve the path and access the data file directly.
S N
- 540
- 1
- 6
- 18
-
But can we use plugins to add a button on the web server UI which upon clicking will open a file selector? Is this possible? – Prinz Piuz Mar 31 '23 at 09:16
-
based on my understanding that is most likely not possible. – S N Mar 31 '23 at 09:44
0
Airflow’s design makes this nearly impossible. DAGs must be available to all Airflow components and these can be very distributed and do not necessarily have network access to one another.
You will instead want to have this upload process live elsewhere and distribute the resulting DAG to your Airflow components yourself, probably through some kind of CI/CD.
Collin McNulty
- 371
- 2
- 6
-
But I have achieved this by using airflow plugins, Basically extending Airflow UI using underlying flask and upon successful upload of file and validations I triggered my dag from plugin – Prinz Piuz Apr 16 '23 at 06:06