Stack Overflow
Stack Overflow

Questions tagged [mwaa]

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed service for Apache Airflow that makes it easy for you to build and manage your workflows in the cloud.

you can easily combine data using any of Apache Airflow’s open source integrations.

use the same familiar Airflow platform as you do today to manage their workflows and now enjoy improved scalability, availability, and security without the operational burden of having to manage the underlying infrastructure

automatically scales capacity up to meet demand and back down to conserve resources and minimize costs

integrated with AWS security services to enable secure access

manages the provisioning and ongoing maintenance of Apache Airflow

https://docs.aws.amazon.com/mwaa/

291 questions
2
votes
1 answer

Integrating DBT with Airflow using MWAA instance

I was recently working on the DBT and Airflow integration on the MWAA instance and I followed this official link and I still struggled. Therefore, thought of posting this. AIRFLOW VERSION: 2.2.2 DBT VERSION: 0.21.1
whatsinthename
  • 1,828
  • 20
  • 59
2
votes
2 answers

AWS Managed Workflows for Apache Airflow UI Link Redirects to Internal Server Error

I keep receiving an "Internal Server Error" redirect when I click the "Open Airflow UI" link in the Amazon MWAA console; it's just a blank white screen with "Internal Server Error" in big black text. I was being successfully redirected to the…
MIKENATOR
  • 21
  • 1
2
votes
2 answers

Upgrade python version in AWS Apache Airflow (MWAA)

I am working with Apache Airflow in AWS (MWAA) and I have deployed a python package that was developed on python 3.8. It was running fine locally, however, on MWAA it starts giving syntax errors probably because the python version in MWAA is 3.7. Is…
Mubashir Ali
  • 69
  • 5
2
votes
1 answer

How can we use airflow-code-editor or simple-dag-editor on AWS airflow MWAA?

The plugins "airflow-code-editor" or "simple-dag-editor" are designed to edit dag directly in airflow interface. Sadly they don't work on MWAA because of a "Read-only file system" error. Is there a way to make them work ?
Hugo
  • 1,195
  • 2
  • 12
  • 36
2
votes
1 answer

Amazon Managed Apache Airflow: Can MWAA pull a docker image from AWS ECR?

We are looking into using MWAA and we already have docker images stored in ECR. Is it possible to pull a docker image in ECR from a DAG running in MWAA ?
wi3o
  • 1,467
  • 3
  • 17
  • 29
2
votes
1 answer

AWS Airflow Add provider - MongoDB

I'm new in the AWS Airflow. I have unmanaged Airflow on AWS ec2 instance which I want to move to the managed one. When I went over the managed Airflow I saw there is missing provider for MongoDB. How can I add it to the "connection type" drop…
Alan Mil
  • 189
  • 1
  • 2
  • 9
2
votes
1 answer

Listing packages in MWAA with a broken scheduler

So, I'm currently working with an Airflow installation via MWAA. I'm having this issue with a broken dependency, specifically: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This…
nick
  • 31
  • 4
2
votes
3 answers

Install custom package (wheel) in MWAA

We are trying to install python package that can be shipped in plugins.zip for MWAA environment and added the reference in requirements.txt as described in the docs but we are still seeing the error file not found. directory structure ├──…
mad_
  • 8,121
  • 2
  • 25
  • 40
2
votes
3 answers

MWAA Apache Airflow DAG error importing EcsOperator

I am trying to deploy an Airfow DAG to MWAA. My requirements.txt: apache-airflow[amazon] == 3.2.0 I import EcsOperator like this: from airflow.contrib.operators.ecs_operator import EcsOperator However, I get this error: Broken DAG:…
sdgfsdh
  • 33,689
  • 26
  • 132
  • 245
2
votes
2 answers

Unable to access newly created Airflow UI MWAA

I am trying to create MWAA as root user and I have all AWS services (s3 and EMR )in North California. MWAA doesn't exist in North California. Hence created this in Oregon. I am creating this in a private network, it also required a new s3 bucket in…
Xi12
  • 939
  • 2
  • 14
  • 27
2
votes
0 answers

Creating airflow variables and connections on MWAA using python requests (aws lambda)

I have a requirement to create airflow variables and connections on Amazon MWAA using a lambda function. Here is my python code to create variables: r = requests.post( mwaa_webserver_hostname, headers={ 'Authorization': mwaa_auth_token, …
Nisman
  • 1,271
  • 2
  • 26
  • 56
2
votes
0 answers

Airflow DAG Dependencies / Orchestration between different workflows

My team is working orchestrating our data pipeline with Airflow. Since our pipeline steps are complex, we were thinking about having different DAGs / workflows, each defined on its own file. Each of the workflows can trigger more than one downstream…
Raquel
  • 89
  • 8
2
votes
0 answers

How does AWS install Airflow in their MWAA service

I have been using AWS MWAA for a while, but recently had some issues with conflicting packages. So I decided to investigate this a little further. For this purpose: I created an MWAA v2.0.2 instance without additional Python packages. The idea here…
Ali Cirik
  • 1,475
  • 13
  • 21
2
votes
1 answer

AWS managed airflow to Databricks SSL error

I am trying to use databricks inbuilt operator provided by airflow such as DatabricksSubmitRunOperator or DatabricksRunNowOperator but i am not able to use it, it gives below error: Attempt 1 API Request to Databricks failed with reason:…
MykG
  • 109
  • 1
  • 11
2
votes
1 answer

MWAA without workers

My current undestanding of MWAA MWAA uses Fargate for Scheduler(s) and Worker(s). MWAA must have at least 1 Scheduler, it must be Fargate, I assume 24/7 endless process (to avoid decomissioning). Scheduler Fargate machine is also running Celery…
VB_
  • 45,112
  • 42
  • 145
  • 293
1 2 3
…
19 20