Questions tagged [airflow-api]
77 questions
1
vote
0 answers
Is thre a way to pass worflow for databricks from airflow with a custom new cluster with DatabricksSubmitRunOperator
I'm traing to find a way to pass worflow for databricks from airflow with a custom new cluster with DatabricksSubmitRunOperator via the json.
If anyoane has an example with all parameters of how this json should look like i will appreciate.
This is…
GLC
- 11
- 2
1
vote
0 answers
Airflow API Version is binded with Airflow Version
Is there any reason for the last stable API be present only in Airflow > 2.4.0. At this moment, the last airflow version in Google Cloud Composer is 2.3.4, and I the API version does not allow dag_run_id field in body request for clearInstances…
Paulo Henrique Zen
- 679
- 2
- 6
- 12
1
vote
1 answer
Access Json data in a task in Airflow
Hi i am triggering a dag with a dag id dag_1 using airflow's REST api and python requests module and in this request I want to send the json data (dic) too. this is how i and sending
import json
import requests
from requests.auth import…
NeedToCodeAgain
- 129
- 2
- 9
1
vote
1 answer
405 method not found PATCH [google cloud composer airflow patch rest api] -JAVA
Airflow patch rest api is not working for JAVA version 11.
Here is the code snippet.
Map params = new LinkedHashMap<>();
params.put("state", "failed");
GoogleCredentials googleCredentials = null;
try {
googleCredentials =…
1
vote
1 answer
How to get the list of DAGS and their scheduled time from cloud composer (airflow environment) in GCP?
I believe there can be a log for this, which stores all the dags and their respective data.
I need to have a list of dags with their schedule time, the expected code can be in any format, python or shell command.
Thanks.
Aniket Ghole
- 192
- 1
- 11
1
vote
2 answers
How to get status of airflow DAGs without Airflow UI
Airflow UI comes very handy to check the status of Airflow DAG progress.
Airflow Rest API is another way to check the status to Airflow DAGs but it requires authentication token.
We can get authentication token from Airflow UI, but if Airflow UI is…
Khilesh Chauhan
- 739
- 1
- 10
- 36
1
vote
1 answer
Airflow unpause dag during manual trigger
I am using Airflow 2.2.2 and have a dag which is scheduled to run every 10 minutes and is paused. I am trying to invoke it manually using airflow client. Dag is not getting unpaused and dagrun is in queued stated. Is it possible to unpause dag using…
Arul
- 143
- 3
- 12
1
vote
2 answers
How to create connector in airflow that is of type external provider (like the google-cloud-plaform) with the airflow REST API
I'm trying to automate creation of connector in airflow by github action, but since it is an external provider, the payload that need to be sent to airflow REST API doesn't work and i didn't find any documentation on how to do it.
So here is the…
Philippe LETAIF
- 21
- 3
1
vote
0 answers
How to retrieve params submitted to airflow dag run (retrieve the json provided during the rest call)
when triggering airflow via REST interface a json with the dag run parameters is provided.
How can i retrieve the json passed to the dag when triggering it?
Example
Having triggered a dag:
curl -X POST…
Anas
- 21
- 3
1
vote
0 answers
Apache Airflow : Cannot trigger a new dag run with role - User
I have recently upgraded my Airflow version from v1.10.6 to v2.2.3(Latest version). I created a user with role - User.
airflow users create \
--role User \
--username DEVUSER \
--firstname DEV \
--lastname USER \
--email…
Bhanu Prakash
- 149
- 3
- 13
1
vote
0 answers
DataprocSubmitJobOperator Fails Intermittent With Zombie
We are using Airflow as orchestrator where it schedule workflow every hour. DataprocSubmitJobOperator is configured to schedule dataproc jobs (it uses spark). Spark sync data from source to target (runs for 50 min and then completes to avoid next…
Coder
- 23
- 4
1
vote
1 answer
How to schedule execution without delay?
I have one problem, I need to launch one DAG every first day of month, but I have one problem, the DAG started on 1 October but executed that day on 1 November, I need that 1 October execute 1 October and 1 November execute 1 November, and not delay…
Guille
- 23
- 2
1
vote
1 answer
Airflow is not loading my configuration file
Hi I'm trying to test out the airflow API, and for that I'm trying to change the default configuration
[api]
auth_backend = airflow.api.auth.backend.deny_all
As guided at…
Jose Pato
- 101
- 8
1
vote
1 answer
Using dagrun.conf on custom operator
I am using airflow 2 stable rest API to trigger a dag. I have created a custom DAG that takes input from MySQL (2 tables) join on key.
and in the body of API, I have to send some parameters like below and these parameters will decide which 2 tables…
Ayush Goyal
- 415
- 4
- 23
0
votes
0 answers
Override/update one of the params through a file in Airflow
How to update the params on the fly?
I want to update 'input_file' param to the path present in file 'temp_file.txt'
import datetime as dt
from airflow import DAG
from airflow.providers.ssh.operators.ssh import SSHOperator
from…
tanviz1992
- 1
- 1