Stack Overflow
Stack Overflow

Questions tagged [airflow-api]

77 questions
0
votes
1 answer

Secure way to utilize AWS SSM parameter store to make API call

I need to write a lambda function which makes an API call (to Airflow) using credentials stored in AWS SSM parameter store. I have been supplied with the key id for the credentials. How can I securely query the credentials and integrate them (again…
Sanchez333
  • 318
  • 3
  • 11
0
votes
2 answers

airflow API get all unpaused/running dags

How do I get a list of all unpaused(running) dags using airflow API? I tried GET /dags endpoints but I did not find a query string to filter paused dags, isn't there something like is_paused query parameter or body parameter perhaps? P.S I'm…
Sannix19
  • 75
  • 1
  • 6
0
votes
2 answers

Get delta of updated records using sqlSensor in AirFlow based on last updated time

I'm trying to use AirFlow SqlSensor to get the updated/inserted new records in a postgresql database. There is a column in this table representing the timestamp of the last update. I want to use SqlSensor in Airflow to fetch the newly updated /…
Mohamed Maherz
  • 41
  • 6
0
votes
0 answers

Airflow Webserver cant fetch log from Worker Celery

I have a problem with Apache Airflow Logging. My Airflow cluster has one Webserver, one Scheduler and five Workers Celery. All Workers working usually but only four Workers can fetch the log, one worker cant fetch the record with error: "Failed to…
Nguyễn Hùng
  • 13
  • 3
0
votes
1 answer

Can I create a Airflow DAG dynamically using REST API?

Is it possible to create a Airflow DAG programmatically, by using just REST API? Background We have a collection of models, each model consists of: A collection of SQL files that need to be run for the model We also keep a JSON file for each model…
AnalystCave.com
  • 4,884
  • 2
  • 22
  • 30
0
votes
1 answer

How to use airflow secret backend with aws secret manager

I have installed the airflow 1.10.15 in standalone server & trying to integrate aws secret manager with it but values are not coming. I have added backend = airflow.contrib.secrets.aws_secrets_manager.SecretsManagerBackend and backend_kwargs =…
Saurabh Srivastava
  • 175
  • 8
0
votes
0 answers

Running multiple dagruns in a second

I am triggering an airflow dag with an api but it is throwing me this response { "detail": "DAGRun with DAG ID: 'working_1' and DAGRun logical date: '2022-11-03 06:35:14+00:00' already exists", "status": 409, "title": "Conflict", "type":…
NeedToCodeAgain
  • 129
  • 2
  • 9
0
votes
0 answers

Airflow - How to use Rest-API with OAUTH

We've set up Airflow 2.3.3 to authenticate users against Azure AD by means of OAuth-2. So our webserver_config.py holds AUTH_TYPE = AUTH_OAUTH and an OAuth provider. Authenticating users in Airflow's web GUI works perfectly fine. Now we'd like to…
dovregubben
  • 364
  • 2
  • 16
0
votes
0 answers

Airflow - Druid Operator is not getting host

I have a question for Druid Operator. I see that this test is successful, but I take a this error. File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 792, in get_adapter raise InvalidSchema(f"No connection adapters…
Emin Can OĞUZ
  • 1
  • 2
0
votes
1 answer

execution_date_gte parameter ignored in Airflow REST API dagRuns call

The execution_date_gte parameter seems to have no effect in below Airflow REST API dagRuns call. curl -X GET 'http://localhost:8080/api/v1/dags/demand_forecast/dagRuns' -H 'Cache-Control: no-cache' -H 'Content-Type: application/json' -d…
user7727736
  • 97
  • 1
  • 8
0
votes
0 answers

Airflow on_success_callback and other types of callbacks not giving proper information on task level

create_volume = PythonOperator( task_id='dummy', python_callable=create_temp_volume, provide_context=True, retries=3, dag=dag, on_success_callback=partial(update_status_test, DAG_ID), on_failure_callback=partial(update_status_test,…
Boss Boss
  • 3
  • 1
  • 5
0
votes
1 answer

What is the structure of the executable transformation script for transform_script of GCSFileTransformOperator?

Currently working on a task in Airflow that requires pre-processing a large csv file using GCSFileTransformOperator. Reading the documentation on the class and its implementation, but don't quite understand how the executable transformation script…
Christine
  • 37
  • 4
0
votes
1 answer

Insert into BigQuery using parameters from web server request json data

In the google function I'm triggering a dag: endpoint = f"api/v1/dags/{dag_id}/dagRuns" request_url = f"{web_server_url}/{endpoint}" json_data = {"conf": {"file_name": file_name}} response = make_composer2_web_server_request( …
Ruslan Lomov
  • 1
  • 2
0
votes
1 answer

Couldn't connect to dpms while creating dataproc using airflow operator

I have a service created for dataproc metastore(in same project as composer's) and trying to use it instead of my hive warehouse. I could successfully run this using gcloud commands but when I am trying to use any airflow operators…
Premchand210
  • 1
  • 2
0
votes
1 answer

Is it safe to delete airflow.db file?

I can't login in the web UI of airflow. I always get this Invalid login error. When I run the airflow users list the users are there. So my option is to delete the airflow.db and run airflow db init again. I want to know if it is safe to do that?
mee
  • 688
  • 8
  • 18
1 2 3
4
5 6