Stack Overflow
Stack Overflow

Questions tagged [mwaa]

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed service for Apache Airflow that makes it easy for you to build and manage your workflows in the cloud.

you can easily combine data using any of Apache Airflow’s open source integrations.

use the same familiar Airflow platform as you do today to manage their workflows and now enjoy improved scalability, availability, and security without the operational burden of having to manage the underlying infrastructure

automatically scales capacity up to meet demand and back down to conserve resources and minimize costs

integrated with AWS security services to enable secure access

manages the provisioning and ongoing maintenance of Apache Airflow

https://docs.aws.amazon.com/mwaa/

291 questions
0
votes
2 answers

EcsCreateClusterOperator import error in MWAA (Apache Airflow)

I am trying to use class EcsCreateClusterOperator in MWAA but I get the following error: Broken DAG: [/usr/local/airflow/dags/etl_basic_ecs/etl_ecs_basic_dag.py] Traceback (most recent call last): File "", line 219, in…
Armando Fa Menor
  • 1
0
votes
0 answers

How to install bash commands inside Amazon Managed Apache Airflow (MWAA)?

I have a DAG with a BashOperator that require mysql cli to dump a database into another like this: dump_command = f""" mysqldump --defaults-file={source_conf} --databases myschema | mysql --defaults-file={target_conf} """ dump_and_load =…
Brenno
  • 1
0
votes
2 answers

Newly installed Airflow providers don't show in Web UI

I'm using MWAA and when I add new providers via requirements.txt they do not show in the "Airflow - Admin > Providers" UI, while the default providers installed on the MWAA instance show just fine. An example: I add…
theNewfelll
  • 19
  • 4
0
votes
1 answer

How to template outside an operator or python callable function

I need to find a way to pull from xcom outside an operator. Currently, I have a list being set and retrieved from the environment variable but im expecting lots of jobs coming through so i foresee a situation where different jobs are setting and…
Musa
  • 99
  • 1
  • 6
0
votes
1 answer

Trigger airflow DAG using post request with parameters

I'm trying to trigger Airflow's DAG and send parameters inside the post request. Usage example - DAG has one task that only prints the number sent inside the trigger request. Example to trigger dag: import base64 import boto3 import…
Amit Nahmias
  • 11
  • 2
0
votes
0 answers

How to upload DAG file into AWS-MWAA-LOCAL?

I am using docker on windows and following this guide (https://github.com/aws/aws-mwaa-local-runner) I was able to install the mwaa-local, but I was not able to upload a dag to UI. I just put my DAG file on aws-mwaa-local-runner\dags folder and…
Wong Chloe
  • 45
  • 9
0
votes
1 answer

Email alerts on failure in Airflow without turning on "Allow less secure apps"

To send email failure alerts, Airflow needed an SMTP username and password for authentication. I am using google mail for that but google treats all apps that use username and password as less secure. So to make it work, I need to enable "Allow less…
hemant A
  • 185
  • 14
0
votes
1 answer

How to use a returned value outside an operator/tasks

I have a dag like the one below and i need to use the value returned from a python operator outside the tasks. How do i achieve this? dag = DAG( dag_id='example_batch_submit_job', schedule_interval=None, start_date=datetime(2022, 7,…
Musa
  • 99
  • 1
  • 6
0
votes
2 answers

All DAGs broken after MWAA update from 2.0.2 to 2.2.2

I am getting the following errors in AWS MWAA UI after I updated from 2.0.2 to 2.2.2 I have exhaustively searched for more details on these errors to no avail from airflow.providers.slack.operators.slack_webhook import…
ab_padfoot
  • 63
  • 1
  • 10
0
votes
1 answer

How do I pass parameters from python operator to batch operator?

I have the below dag which is triggered via cli with a conf parameter for job_id. I have successfully triggered the dag, and can retrieve the job_id but im not sure how to now pass this parameter to the batchoperator? dag = DAG( …
Musa
  • 99
  • 1
  • 6
0
votes
1 answer

importing python dask package on aws MWAA airflow not working

I am trying to import dask package on AWS MWAA (airflow) using requirements.txt, by adding "dask==2022.6.0" line to the file. Not only the import does not work, the other packages are not imported anymore. It is like MWAA does not recognize…
Khaled
  • 1
0
votes
1 answer

Problems connecting Redshift to Airflow (MWAA)

I am learning airflow and as a practice exercise im trying to create a table at Redshift through an airflow dag at MWAA. I create the connection to Redshift at the UI (specifying host,port, etc) and run the following dag, but it fails at the…
d4t4beaver
  • 1
  • 2
0
votes
1 answer

Airflow Passing date Xcom variable to a sql file using RedshiftSQLOperator

I want to pass the date value (from a variable) as partition column in sql file. The variable is not getting evaluated. The variable is stored in Xcom. Is there any better way to do . Rendered template: TO…
user3858193
  • 1,320
  • 5
  • 18
  • 50
0
votes
1 answer

Passing Jinja variables in airflow params dict

I am trying to use templated params dict in SnwoflakeOperator. I have 2 tasks one to create a S3 object and then delete and replace the content of the table with s3 object content execution_date = "{{ execution_date.int_timestamp | int }}" # built…
mad_
  • 8,121
  • 2
  • 25
  • 40
0
votes
1 answer

Apache Airflow (MWAA) DAG that can send a file from PythonOperator to SSHOperator

I'm looking to create a DAG that can run two tasks. The ideal workflow would be to create a PEM key file then SSH using that newly created file. One task would be with the PythonOperator to call AWS Secrets Manager through the SecretsManagerHook.…
rk92
  • 551
  • 4
  • 21
1 2 3
…
19 20