Stack Overflow
Stack Overflow

Questions tagged [mwaa]

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed service for Apache Airflow that makes it easy for you to build and manage your workflows in the cloud.

you can easily combine data using any of Apache Airflow’s open source integrations.

use the same familiar Airflow platform as you do today to manage their workflows and now enjoy improved scalability, availability, and security without the operational burden of having to manage the underlying infrastructure

automatically scales capacity up to meet demand and back down to conserve resources and minimize costs

integrated with AWS security services to enable secure access

manages the provisioning and ongoing maintenance of Apache Airflow

https://docs.aws.amazon.com/mwaa/

291 questions
0
votes
1 answer

Apache Airflow error run id already exists

Getting this error even after passing custom UUID as run_id to airflow dagrun.: run id already exists In order to create multiple DagRun within one second, One should set parameter "replace_microseconds" to "false" according to documentation. But…
Paresh
  • 3
  • 2
0
votes
0 answers

Use AwsHook with refreshable credentials Airflow

I am creating an EMR client using IAM credentials obtained via STS assume role. session_name = 'example-role' client = boto3.client('sts',region_name='eu-west-1',endpoint_url='https://sts.eu-west-1.amazonaws.com') response =…
15b3m
  • 15
  • 2
0
votes
1 answer

Airflow (MWAA) tasks receiving SIGTERM but task is externally set to success

We face a lot of our Airflow (MWAA) tasks receiving SIGTERM: [2022-10-06 06:23:48,347] {{logging_mixin.py:104}} INFO - [2022-10-06 06:23:48,347] {{local_task_job.py:188}} WARNING - State of this instance has been externally set to success.…
val
  • 329
  • 2
  • 16
0
votes
2 answers

MWAA Custom Operator (via plugins) Inheritance from amazon providers installed via pip throws Broken plugin: ... No module named

Im trying to work with custom operators in managed amazon airflow (MWAA). the tick is that my custom operator relaying on amazon provider package that installed via pip upon setup (requirements.txt) The operator I'm inheritance from is…
Benny Elgazar
  • 243
  • 2
  • 9
0
votes
0 answers

Managed Airflow environment creation fails

I am creating managed airflow environment on AWS, and it keeps hitting the status 'Create failed' and this is what the error message says: error code: INCORRECT_CONFIGURATION You may need to check the execution role permissions policy for your…
Harsha Sridhara
  • 3
  • 2
0
votes
0 answers

Where is POD_TEMPLATE FILE Configured in AWS MWAA and how to to Edit it?

Im new to AWS MWAA. i just need to have idea where pod_template_file created in AWS Airflow and where we managing the same and how to modify the same?
Anesly Manoj
  • 3
  • 3
0
votes
1 answer

IAM policy to allow MWAA to use built in AWS KMS key specifies key as "NotResource" to work correctly

I'm having a little trouble getting my head around some IAM policy syntax to do with MWAA and KMS, and was wondering if anyone may be able to help me understand please. From this…
Scott
  • 45
  • 5
0
votes
1 answer

Send reply-to param in airflow emailoperator

I am sending mail using airflow emailOperator. Is there any way to send "reply-to" parameter also. i tried default_param={'key':'value'} but throwing exception as invalid parameter. I read this…
Kundan Kumar
  • 33
  • 7
0
votes
0 answers

Error while deploying MWAA using Cloudformation

I am getting the following error while deploying Managed Workflow using Apache Airflow using Cloudformation. The error doesnt give out much information and hence difficult to debug 2022-09-16 15:25:42 UTC+0530 MwaaEnvironment CREATE_FAILED …
Rohit Anil
  • 236
  • 1
  • 11
0
votes
1 answer

Files created in airflow task deleted automatically or not

In my airflow task, I am creating a file using open() method in airflow dag and writing records into it. Then sending it with a mail within same task. Will it get deleted automatically or will exists into the dag? filename =…
Kundan Kumar
  • 33
  • 7
0
votes
1 answer

Aws MWAA Airflow rest api

Iam trying to hit the api provided in the airflow ui on aws console. I tried to access the web server but when i try to get the list of dags iam getting unauthorized error. Do we need any role or permission to access rest api and get the expected…
kaushik
  • 9
  • 1
0
votes
0 answers

Amazon MWAA - module not found [Duplicated]

I'm setting up a new Apache Airflow environment using MWAA with a requirements.txt file. (AirFlow version 2.2.2) In the requirements.txt file, I am specifically stating a certain version of Amazon provider…
Mark D
  • 1
  • 2
0
votes
0 answers

Snowpipe error : Failed to access remote file: access denied. Please check your credentials

We are using snowpipe to upload files from AWS S3 to snowflake via storage integration. The auto ingestion is set True for this. Files are placed/ uploaded into S3 bucket by MWAA script. The problem which is occurring here, that sometimes data is…
user2388773
  • 155
  • 8
0
votes
0 answers

I keep getting a connection timed out when connecting Airflow to EMR cluster

I am trying to connect a Managed Airflow instance to an already provisioned EMR cluster using the following code: import os from datetime import datetime from airflow import DAG from airflow.models.baseoperator import chain from airflow.models…
15b3m
  • 15
  • 2
0
votes
3 answers

Having issues setting the start_date in Airflow / MWAA

Problem: The start date of my DAG is not being set properly, can anyone tell me why? Here is sample code: default_args = { "owner": "hello", "email_on_failure": "false", "retries": 1, "retry_delay": timedelta(minutes=1), …
user19138502
  • 43
  • 1
  • 7
1 2 3
…
19 20