Stack Overflow
Stack Overflow

Questions tagged [mwaa]

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed service for Apache Airflow that makes it easy for you to build and manage your workflows in the cloud.

you can easily combine data using any of Apache Airflow’s open source integrations.

use the same familiar Airflow platform as you do today to manage their workflows and now enjoy improved scalability, availability, and security without the operational burden of having to manage the underlying infrastructure

automatically scales capacity up to meet demand and back down to conserve resources and minimize costs

integrated with AWS security services to enable secure access

manages the provisioning and ongoing maintenance of Apache Airflow

https://docs.aws.amazon.com/mwaa/

291 questions
3
votes
3 answers

AWS MWAA Pricing

It is not clear to me if the aws mwaa documentation says the following "With Amazon Managed Workflows for Apache Airflow (MWAA) you pay only for what you use. There are no minimum fees or upfront commitments. You pay for the time your Airflow…
datahack
  • 477
  • 1
  • 11
  • 32
3
votes
3 answers

Can't create a new working MWAA environment

In the past two days, we can't create a new working MWAA environment. We started with Terraform - after apply, the environment is indicated as "Available" in the console, but when I click on the "Open UI" link, the UI never comes up. Then we…
Tony Jiang
  • 442
  • 5
  • 16
3
votes
5 answers

Amazon Managed Airflow (MWAA) import custom plugins

I'm setting up an AWS MWAA instance and I have a problem with import custom plugins. My local project structure looks like this: airflow-project ├── dags │ └── dag1.py └── plugins ├── __init__.py └── operators ├── __init__.py …
Tal Meridor
  • 31
  • 1
  • 2
3
votes
1 answer

Cannot configure docker-compose as an interpreter in PyCharm

I have a problem with configuration of Docker-compose as a remote interpreter in PyCharm. Inside of "Services" section I can select only "services" or "networks". I am using this docker-compose file:…
bambuste
  • 147
  • 2
  • 9
3
votes
2 answers

MWAA: install python requirements behind proxy

we've launched a private MWAA environment. We are able to access the UI, but we're having some trouble installing our python requirements. MWAA picks up the requirements file from S3, but runs into a timeout when trying to install the python…
dovregubben
  • 364
  • 2
  • 16
3
votes
0 answers

What is the best way to run jar from MWAA?

I want to run EMR spark job which will output data to S3, when job completed terminate cluster and submit custom jar which will import data to Redshift I am deploying all my jar files to S3 folder For EMR I am using Airflow EMR/Livy operators to…
Grish
  • 93
  • 6
3
votes
3 answers

AWS Airflow v2.0.2 doesn't show Google Cloud connection type

I want to load data from Google Storage to S3 To do this I want to use GoogleCloudStorageToS3Operator, which requires gcp_conn_id So, I need to set up Google Cloud connection type To do this, I added apache-airflow[google]==2.0.2 to…
Grish
  • 93
  • 6
3
votes
1 answer

AWS MWAA/Apache airflow: how to debug on_failure_callback itself

Have a dag like this: import os from datetime import timedelta from xxx import on_failure_opsgenie from airflow import DAG from airflow.operators.python import PythonOperator from airflow.utils.dates import days_ago DAG_ID =…
Tommy
  • 12,588
  • 14
  • 59
  • 110
3
votes
2 answers

Use default vs separate VPC for AWS MWAA

I am currently setting up AWS MWAA. I will be using Apache Airflow for performing aggregations on an AWS RDS database that is in the default VPC. Should I create a new VPC for AWS MWAA or can I use the default VPC? What are the advantages of having…
Developer
  • 2,113
  • 2
  • 18
  • 26
3
votes
1 answer

Accessing Airflow REST API in AWS Managed Workflows?

I have Airflow running in AWS MWAA, I would like to access REST API and there are 2 ways to do this but doesn't seem to work for me. Overriding api.auth_backend. This used to work and now AWS MWAA won't allow you to add this, it is consider as…
user15364819
  • 61
  • 5
3
votes
1 answer

AWS MWAA (Managed Apache Airflow); Programmatically enable DAGs

We are using AWS MWAA. We add our DAG.py files to our S3 bucket programatically. They then show up in the UI. However, they are "OFF" and you must click the "ON" button to start them. EDIT: Also we may sometimes want to turn a DAG that's ON to OFF…
Tommy
  • 12,588
  • 14
  • 59
  • 110
3
votes
1 answer

AWS Managed Airflow - how to restart scheduler?

I have a problem parsing DAG with error: Broken DAG: [/usr/local/airflow/dags/test.py] No module named 'airflow.providers' I added apache-airflow-providers-databricks to requirements.txt, and see from the log that: Successfully installed…
Keren Kapach
  • 31
  • 1
  • 2
3
votes
0 answers

What is the best practice for multi-tenancy with AWS document DB and AWS managed workflows

What is the best practice for multi-tenancy with AWS document DB and AWS managed workflows? Different VPCs with vpc peering: Is it better to create one cluster for several tenants and separate tenants with Role-Based Access Control? So the…
Maria Dorohin
  • 355
  • 4
  • 17
3
votes
1 answer

How do I best integrate AWS MWAA and Docker?

Our data pipeline setup currently makes use of Docker containers running on EC2 instances and Apache Airflow as the orchestration tool. We're trying to move to MWAA for the advantages associated with a managed service(scalability, maintenance,…
roni4743
  • 97
  • 1
  • 7
3
votes
1 answer

Airflow schedule_interval and start_date to get it to always fire the next interval

How can I configure airflow (mwaa) so that it will fire at the same time (6am PST) every day regards of when the dag is deployed? I have tried what makes sense to me: set the schedule_interval to 0 6 * * *. set the start date to: now =…
cosbor11
  • 14,709
  • 10
  • 54
  • 69
1 2
3
…
19 20