Stack Overflow
Stack Overflow

Questions tagged [mwaa]

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed service for Apache Airflow that makes it easy for you to build and manage your workflows in the cloud.

you can easily combine data using any of Apache Airflow’s open source integrations.

use the same familiar Airflow platform as you do today to manage their workflows and now enjoy improved scalability, availability, and security without the operational burden of having to manage the underlying infrastructure

automatically scales capacity up to meet demand and back down to conserve resources and minimize costs

integrated with AWS security services to enable secure access

manages the provisioning and ongoing maintenance of Apache Airflow

https://docs.aws.amazon.com/mwaa/

291 questions
0
votes
2 answers

ModuleNotFoundError: No module named 'snowflake' in Apache Airflow DAG despite being included in requirements.txt

I have been struggling with an issue in my Apache Airflow DAG for some time now. I am trying to use the snowflake-connector-python module in my DAG, and it is included in my requirements.txt file. However, when I try to run the DAG, I keep getting…
Nyk Casseano
  • 5
  • 2
0
votes
0 answers

How to correctly manage thousands of ETL replication objects

How can I efficiently manage the replication of thousands of objects from Salesforce in MWAA using separate DAGs per object, prioritizing some objects over others and avoiding restarting the entire process in case of failure? Context: I want to…
Juanjo
  • 79
  • 1
  • 8
0
votes
1 answer

AWS Cli - MWAA - Create environment with configuration options

I'm creating MWAA enviornments through AWS Cli with the create-environment function. I can create the environment without any problems but now I'm trying to add a configuration option so I can use AWS Secret Manager as the backend…
Pedro Alves
  • 1,004
  • 1
  • 21
  • 47
0
votes
2 answers

XCom Value Not showing in Airflow MWAA

I'm running following script in MWAA and my local env. from airflow import DAG, XComArg from airflow.providers.amazon.aws.operators.redshift_sql import RedshiftSQLOperator from airflow.models.connection import Connection from…
AIViz
  • 82
  • 9
0
votes
2 answers

Number of concurrent tasks run in AWS managed Airflow is not linearly dependant on the number of Workers

AWS provides to run Airflow as a service. It has 3 different environments in which it can be run, namely Small, Medium and Large. These environments have different configurations of CPU and RAM. I used the Large config which has 4vCPUs and 8GB RAM…
HENSEL WILSON GOVEAS
  • 21
  • 5
0
votes
0 answers

MWAA Airflow 2.4.3 Slack Connection Not showing

I've got following modules in my requirements file. However, slack isn't showing up when I create a new connection via UI. In my local environment it is working just fine. I have tried recreating the environment but no luck. I am using MWAA Airflow…
AIViz
  • 82
  • 9
0
votes
0 answers

Slack connection not showing up in MWAA UI

Currently, I've got an MWAA env setup. I am trying to create a slack connector via UI but for some reason it's not showing up. I've tested it locally it is working just fine. My requirements.txt file contains --constraint "s3://
AIViz
  • 82
  • 9
0
votes
1 answer

MWAA 2.4.3 - AWSGlueJobOperator failed

I am trying to update my AWS MWAA to Airflow 2.4.3 The DAGs I am using in 2.2.2 are breaking in the new 2.4.3 environment. One such DAG is using AWSGlueJobOperator glue_task = AWSGlueJobOperator( task_id="TS-Learn-Source-To-Stage", …
ab_padfoot
  • 63
  • 1
  • 10
0
votes
2 answers

MWAA Redshift Connection Timeout

I am trying to setup Airflow on Managed Apache Airflow. Everything seems to be working fine except for my AWS Redshift connection. I am using the Connections tab on UI and editing redshift_default with my values. It's working fine locally however…
AIViz
  • 82
  • 9
0
votes
4 answers

MWAA SSH into own instance

I am trying to ssh into my own AWS MWAA instance in order to install some system dependencies. I'm coming from GCP so this is a bit different for me. I can't find the exact EC2 it is hosted on. Or derive the the IP for some reason. And I don't think…
Mojo713
  • 33
  • 2
  • 7
0
votes
1 answer

MWAA load custom file info to DAG

I'm trying to use a file in a DAG. The codes I want to use is basically this: conf_device_info = OmegaConf.load(f"./config/{dag_name}/config_dtype.json") and my bucket is currently like this: my-bucket --/ dags -- /config …
Hoon
  • 377
  • 1
  • 4
  • 17
0
votes
0 answers

MWAA Trigger DAG Issue with POST request

I have a problem when I try to execute multiple Tasks within MWAA using POST Requests. I have been using mw1.small tier of MWAA and I schedule around 3 tasks per minute with EventBridge and Lambda. When I see my results I find that some tasks are…
krozmok
  • 1
  • 1
0
votes
1 answer

Connection Time out when accessing MWAA UI

I created MWAA environment on the main default VPC where every service is running .I used the (private network) option to access the webserver. When accessing the airflow UI using my user the connection times-out and the site is not reached. The…
Mohamed Ashraf
  • 21
  • 1
0
votes
1 answer

How to enforce max active run = 1 for a group of DAGs in Airflow?

I have a group of DAGs and I only want one of them to run at any given time. ExternalTaskSensor will not work if I trigger a backfill job for one of them for a very old date. I am aware of pool and priority weights method. Another approach could be…
Dark Matter
  • 373
  • 6
  • 18
0
votes
0 answers

Amazon MWAA not capturing the state of latest retry

I am using Amazon MWAA. I have created a DAG with multiple Glue jobs. Few of the glue jobs are failing in first try but succeeding in 2nd/3rd retries. But MWAA capturing state of first try and making my DAG failed. Is there any way that MWAA wait…
connecttopawan
  • 208
  • 3
  • 14
1 2 3
…
19 20