Stack Overflow
Stack Overflow

Questions tagged [airflow-2.x]

Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks. Use this tag for questions about version 2+ of Airflow. Use the more generic [airflow] tag on all Airflow questions, and only add this one if your question is version-specific.

690 questions
0
votes
1 answer

Error response from daemon: dockerfile parse error line: unknown instruction: FUNCTION

I am trying to build docker image from https://github.com/apache/airflow/blob/2.3.2/Dockerfile This file has a function method defination However getting below error Error response from daemon: dockerfile parse error line 76: unknown instruction:…
Sachin
  • 3,424
  • 3
  • 21
  • 42
0
votes
1 answer

Airflow:2.3.0:MsSqlConnect:Adaptive Server is unavailable or does not exist

Requirement: Connect to MSSQL Server from Airflow Airflow Version - 2.3.0 Gave URI as the connection in AWS SecretManager as…
Kar
  • 790
  • 13
  • 36
0
votes
1 answer

Facing task Timeout Error while dag parsing in Airflow version 2.2.5

I am hitting the task timeout error with Airflow Version 2.2.5/Composer 2.0.15. The same code is running absolutely fine in Airflow version2.2.3 /Composer Version 1.18.0 Error Message : Broken DAG: [/home/airflow/gcs/dags/test_dag.py] Traceback…
codninja0908
  • 497
  • 8
  • 29
0
votes
2 answers

Apache Airflow Webserver Log In Failed

I realise this is a duplicate of thousands of posts across the internet, but I have only found solutions for version <2.0, and those solutions do not apply to >2.0. Basically, I have installed Apache Airflow on an Ubuntu system. Everything installed…
Vinny Pem
  • 57
  • 1
  • 6
0
votes
1 answer

Upload GCS File to Google Drive (Airflow)

I am using airflow.providers.google.suite.transfers.gcs_to_gdrive.GCSToGoogleDriveOperator to upload a file from GCS to Google Drive. Getting 403 Forbidden with reason "insufficientPermissions" This is the logging. Not sure where and what the…
anu
  • 41
  • 3
0
votes
2 answers

I have a ExternalTaskSensor ERROR,AirflowSensorTimeout

I have a ExternalTaskSensor ERROR, first : DAG A.task1 tigger DAG B.task1, second: DAG A.task3 wait until B.task2 success. so I use ExternalTaskSensor , pic follow: [enter image description here][1] ExternalTaskSensor…
john michael
  • 1
  • 1
0
votes
1 answer

Retrive a Xcomm value and pass it to spark _steps in EMR operator, Airflow

I am trying to retrieve value from a python operator and pass it to "EMR create job" and "add steps operator". How do I pass this value in spark_steps, I have used task_instance to access it but it didn't work, can someone please tell me how to…
Xi12
  • 939
  • 2
  • 14
  • 27
0
votes
0 answers

RunJobFlow operation: An instance group must have at least one instance

I am writing code to run a psyaprk job using AIrflow 2.0.2,and I only need to start the master nodes, no core nodes My JOB_FLOW_OVERRIDES is throwing an error saying: An instance group must have at least one instance. I already have InstanceCount=…
Xi12
  • 939
  • 2
  • 14
  • 27
0
votes
1 answer

how to access s3 bucket in airflow without giving connection string via admin -> connection UI

I would like the connect to aws s3 without making use of the Admin-> configuration UI of Airflow. Can any one ,please let me know is there a way to do so in Apache Airflow?
Nikitha Singh
  • 11
  • 5
0
votes
0 answers

Install and managed different versions of custom whl file in google's cloud composer environment

I am starting out with Google Cloud Composer environment. I want to organize my projects as follow: dags/ ├── project-1 | ├── dist | | ├── my_custom-0.0.1-py3-none-any.whl | ├── src | | ├── project_1_dag.py | ├── config | | ├──…
kshaikh
  • 41
  • 1
  • 8
0
votes
0 answers

Airflow - Adding a task before tasks in loop that run dependent on the previous task

I'm trying to get my task dependencies correct. So far I have 8 tasks that are dependent on the previous task which is great, but then I want to have a task run before these tasks first, but I can't seem to get it correct. In the screenshot you'll…
KristiLuna
  • 1,601
  • 2
  • 18
  • 52
0
votes
1 answer

Polling multiple SQS messages using Airflow SQSSensor

I am using this SQSSensoe settings to poll messages fetch_sqs_message = SQSSensor( task_id="...", sqs_queue="...", aws_conn_id="aws_default", max_messages=10, wait_time_seconds=30, …
Kei
  • 611
  • 2
  • 11
  • 24
0
votes
0 answers

Airflow - Deleting the logs reference in the database deletes the logs in the persistent volume claim?

Hello good afternoon I have encountered a curious occurrence in Airflow 2.3.0 I have a pipeline that is in charge of deleting the logs in the airflow database (I have the BBDD running on Postgress on AWS). This is the pipe: from datetime import…
Tau n Ro
  • 108
  • 8
0
votes
1 answer

How do I enable supplied backend authentication in the Airflow 2.1.0 version?

I have installed Airflow 2.1.0 version recently. But unable to find an option to enable airflow login authentication with supplied backend like in older versions. Can you anyone help me in providing steps to achieve this?
Kartheek
  • 1
0
votes
1 answer

Airflow Task triggered manually but remains in queued state

I am using Airflow 2.3.1 and running with Local Executor against MS SQL Server as metadata db. I am trying to execute a dag manually, it shows as queued but nothing happens. There is no other tasks running when this dag is triggered. When I hover on…
MKG
  • 23
  • 5
1 2 3
…
45 46