Stack Overflow
Stack Overflow

Questions tagged [mwaa]

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed service for Apache Airflow that makes it easy for you to build and manage your workflows in the cloud.

you can easily combine data using any of Apache Airflow’s open source integrations.

use the same familiar Airflow platform as you do today to manage their workflows and now enjoy improved scalability, availability, and security without the operational burden of having to manage the underlying infrastructure

automatically scales capacity up to meet demand and back down to conserve resources and minimize costs

integrated with AWS security services to enable secure access

manages the provisioning and ongoing maintenance of Apache Airflow

https://docs.aws.amazon.com/mwaa/

291 questions
0
votes
1 answer

Is it possible to sync an azure repo with MWAA (Amazon Workflows for Apache Airflow)?

I have set up a private MWAA instance in AWS. It has set up a bucket that stores DAGs in S3. I've created a private repository in Azure DevOps and have set up a role that can access this bucket. With Azure-Pipelines is it possible to sync the entire…
Yanzal
  • 113
  • 1
  • 8
0
votes
1 answer

Remote logging issue with AWS mwaa

I was testing remote logging to the s3 bucket path to send airflow task logs. Previously it was working fine. We were able to see logs in airflow UI when there was no environment variable set for remote_logging but after setting…
Shubhank Gupta
  • 705
  • 2
  • 10
  • 27
0
votes
1 answer

Add an EC2 instance as a worker to Amazon MWAA

I am currently using Amazon MWAA as my Airflow. I want to have 2 types of workers nodes but currently MWAA doesn't support it. I want to have: High Compute Optimized CPU workers GPU workers I want to create different queues for both the worker…
Manikanth Reddy
  • 31
  • 6
0
votes
0 answers

Private MWAA - Snowflake Connection Issue - Amazon Managed Workflows for Apache Airflow

I set up an private Airflow environment in AWS -v2.2.2-. Environment and plugins are up and running, I want to connect to Snowflake but I am getting the error below . -whl files in plugins.zip using…
chook
  • 11
  • 1
0
votes
1 answer

Is it possible to integrate Control-M with MWAA Airflow to trigger and monitor jobs?

Pretty new to these orchestration tools. Is it possible to integrate Control-M and mwaa airflow to trigger/ track jobs? Couldn't find much information regarding the same.
Kya Senpai
  • 1
0
votes
0 answers

MWAA DeprecationWarning (AWS)

I am using MWAA airflow environment in AWS. I am trying to install the mysql provider package through the requirement.txt file and it contains apache-airflow-providers-mysql Before its working fine but now am getting the error…
Kalaiarasu M
  • 31
  • 3
0
votes
0 answers

Multiple remote logging in airflow

How can I set the multiple remote logging destinations in airflow? Is it possible to do the same in aws MWAA? By default MWAA sends the logs to cloudwatch but I want to store the copy of logs to s3 bucket as well. I tried remote_base_log_folder =…
Bibek Paudyal
  • 1
  • 1
  • 1
0
votes
0 answers

Airflow (2.0.2) Snowflake TypeError: snowflake_to_pandas() missing 1 required positional argument: 'query_test'

I am running MWAA on local docker with Airflow2.0.2, and I am trying to get data from snowflake and store into a local folder and I am getting the below "missing 1 required positional ". How to rectify this error? I have checked arguments to…
Xi12
  • 939
  • 2
  • 14
  • 27
0
votes
0 answers

Getting DAG import errors in Apache Airflow metrics but no such error is visible on UI

I am using AWS MWAA (Managed Workflow for Apache Airflow) and setting up monitoring for the same. Airflow Version: v2.2.2 Metric: dag_processing.import_errors My airflow setup is being used by multiple teams to execute their DAGs and so I wanted to…
Rishabh Sharma
  • 747
  • 5
  • 9
0
votes
1 answer

Amazon Managed Airflow 2.2.2 Fails To Run Tasks From The CLI With DeprecationWarning

Context I am upgrading our system from Amazon Managed Airflow 2.0.2 to 2.2.2. I am running into a situation where I can run DAGs in the UI but if I try to run them from the API I'm…
AlexLordThorsen
  • 8,057
  • 5
  • 48
  • 103
0
votes
0 answers

Is there a way to read in an SQL file for an airflow Dag on MWAA in a Python Operator?

I'm trying to simplify my workflow by reading in SQL files that contain the query I would like to run. Within the Python Operator is a snowflake operator that gets executed and then all of this information is passed on to an API that is setup to…
RuSXSnIpERXx
  • 67
  • 4
0
votes
1 answer

Unexpected Jinja Template Behaviour in Custom Airflow Operator

I have made a custom sensor in Airflow which inherits BashSensor. Sensor : class MySensor(BashSensor): def __init__(self, time, **kwargs): # {{ ts }} is passed as time in the DAG self.time = time cmd = f"java some-other-stuff…
Dark Matter
  • 373
  • 6
  • 18
0
votes
0 answers

MWAA EKSPodOperator error when assuming role

we have the following basic EKS Operator on MWAA (Airflow version 2.2.2) start_pod = EKSPodOperator( aws_conn_id="eks-connection", task_id='start_pod', namespace="airflow", cluster_name="eks-data-stg", …
jojo_Berlin
  • 673
  • 1
  • 4
  • 19
0
votes
0 answers

ModuleNotFoundError: No module named 'snowflake' in MWAA Airflow Version 2.2.2

I am using Python snowflake.connector library to connect to snowflake from MWAA Airflow. The code is working fine in Airflow version 2.0.2. Now I am upgrading MWAA Airflow version to 2.2.2 and trying to execute the same code to connect to snowflake.…
Rohit Londhe
  • 1
0
votes
1 answer

Airflow DagRunAlreadyExists even after providing the custom run id and execution date

I am Getting DagRunAlreadyExists exception even after providing the custom run id and execution date. This occurs when there are multiple request within a second. Here is the MWAA CLI call def get_unique_key(): from datetime import datetime …
Ankur Vyas
  • 118
  • 9
1 2 3
…
19 20