Stack Overflow
Stack Overflow

Questions tagged [ariflow]

6 questions
5
votes
2 answers

Kubernetespodoperator how to use cmds or cmds and arguments to run multiple commands

I'm using GCP composer to run an algorithm and at the end of the stream I want to run a task that will perform several operations copying and deleting files and folders from a volume to a bucket I'm trying to perform these copying and deleting…
Amit Lipman
  • 687
  • 7
  • 19
1
vote
0 answers

AirFlow SIGTERM While EMR Job is Running

I have an AriFlow DAG where each step is an AWS EMR task. Once AirFlow reaches one of the steps, it sends the SIGTERM signal as the following {emr_step.py:73} INFO - Poking step XXXXXXXX on cluster XXXXXXXX {emr_base.py:66} INFO - Job flow currently…
A.M.
  • 1,757
  • 5
  • 22
  • 41
0
votes
1 answer

How to dynamically create multiple scheduled dags which runs on specifc time only once in airflow

I have been looking at the airflow documentation for creating a scheduled dag run. My use case is i want to create & schedule a dag run on a specified time only once. This time value i will be getting from datetime column from a database table. For…
Rohit sai
  • 119
  • 1
  • 10
0
votes
1 answer

Can't create Cloud Composer environment on GCP

I'm trying to create a Cloud Composer environment to run Airflow on GCP. However, I keep getting this error: CREATE operation on this environment failed x minutes ago with the following error message: Composer Backend timed out. Currently running…
Dudes
  • 75
  • 6
0
votes
1 answer

Is there a way to parametrize BigQueryOperators in a Dag with Airflow?

So my goal is to create a Dag with BigQueryOperators that I can send in Airflow with a parametrized destination table in my SQL. I checked a lot of topics about how to send parameters to PythonOperators in order to call them with a --conf in Airflow…
Vladimir Suvorov
  • 13
  • 1
0
votes
1 answer

Why can't I get to the UI for AWS Airflow using webtoken?

In AWS Console, I did the following: Created an S3 bucket & key: s3://my-airflow and s3://my-airflow/dags Setup an Airflow Environment. Created and attached a Service role as described here:…
cosbor11
  • 14,709
  • 10
  • 54
  • 69