Stack Overflow
Stack Overflow

Questions tagged [mwaa]

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed service for Apache Airflow that makes it easy for you to build and manage your workflows in the cloud.

you can easily combine data using any of Apache Airflow’s open source integrations.

use the same familiar Airflow platform as you do today to manage their workflows and now enjoy improved scalability, availability, and security without the operational burden of having to manage the underlying infrastructure

automatically scales capacity up to meet demand and back down to conserve resources and minimize costs

integrated with AWS security services to enable secure access

manages the provisioning and ongoing maintenance of Apache Airflow

https://docs.aws.amazon.com/mwaa/

291 questions
4
votes
2 answers

AWS MWAA; How to install specific version of AWS CLI V2

So, we can see from this fun issue: https://github.com/aws/aws-cli/issues/4947 that AWS CLI 2 is never going to be pushed to pip. In AWS MWAA (Managed AIrflow), we are not given the ability to specify apt-get level installs. We get a…
Tommy
  • 12,588
  • 14
  • 59
  • 110
4
votes
1 answer

Selenium and Chromium on AWS-MWAA

I'm currently using AWS Apache Airflow service (MWAA) and I've been needing to run some web scraping code using selenium, I did manage to add "chromedriver" and "selenium" to the dependencies of the project, but I've been struggling to add chromium…
Lucas
  • 161
  • 1
  • 8
4
votes
0 answers

Airflow jobs on MWAA fail with no Log messages

I have been using Airflow on AWS(MWAA) for a couple of months now and I noticed that occasionally some Airflow tasks fail with no discernable reason and with no log messages in Cloudwatch. I often have to clear and retry the tasks multiple times for…
roni4743
  • 97
  • 1
  • 7
4
votes
4 answers

No module named ____ error in AWS MWAA even with requirements.txt

I've uploaded my requirementst.txt to a dag and referenced it when creating my Airflow Environment. Here is what my requirements.txt looks…
semiflex
  • 1,176
  • 3
  • 25
  • 44
4
votes
2 answers

Is it possible to configure AWS CodeArtifact PyPi repository to be used in AWS MWAA?

AWS MWAA (Managed Workflows for Apache Airflow) is relatively new service provided by AWS. When configuring the MWAA environment, it is possible to provide custom requirements.txt file, which is used to install additional Python packages in that…
Peter Jakubčo
  • 83
  • 1
  • 6
4
votes
2 answers

How can I enable the API in AWS Managed Workflows for Apache Airflow?

I'm testing the waters for running Apache Airflow on AWS through the Managed Workflows for Apache Airflow (MWAA). The version of Airflow that AWS have deployed and are managing for me is 1.10.12. When I try to access the v1 REST API at…
urig
  • 16,016
  • 26
  • 115
  • 184
4
votes
1 answer

Broken DAG: No module named 'psycopg2' when using AWS Airflow Managed Service

I'm trying to use the PostgresHook inside a DAG in AWS Airflow Managed Service as the following way: from airflow.hooks.postgres_hook import PostgresHook The Airflow version that uses this service is 1.10.12, but when I upload this DAG the Airflow…
Kevin Gomez
  • 43
  • 5
3
votes
2 answers

MWAA Airflow job getting SCRAM error when connecting to postgres

I'm trying to query postgres from an MWAA instance of airflow. I'm not sure if there is a conflict due to airflow itself having a different version of postgres for its metadata or what, but I get this error when connecting to postgres: File…
RagePwn
  • 411
  • 2
  • 8
  • 22
3
votes
0 answers

Airflow Not triggering sla_miss_callback for my case

I have followed the documentation and previous stackoverflow links eg : Airflow sla_miss_callback function not triggering Still I am not able to trigger sla_miss_callback for my case. I have a top level DAG(DAG X1) that has scheduler interval of : 0…
Mayank Kataruka
  • 395
  • 1
  • 2
  • 11
3
votes
1 answer

How to pass parameters to MWAA (airflow) dag from lambda?

I need help in passing the arguments (conf params) to mwaa (airflow) from lambda. Lmabda is used to trigger the dag on sqs event. The dag runs fine without command line params. import boto3 import http.client import base64 import ast mwaa_env_name…
user3858193
  • 1,320
  • 5
  • 18
  • 50
3
votes
0 answers

How to connect MWAA CLI that is hosted in a public subnet

I am trying to invoke airflow Cli from my local terminal and I have followed these instruction https://dnx.solutions/how-to-use-apache-airflow-cli-with-amazon-mwaa/ and created cli_token and executed curl \ --request POST…
Xi12
  • 939
  • 2
  • 14
  • 27
3
votes
1 answer

Set PYTHONPATH in MWAA

I'm trying to use local module inside a dag on MWAA. The folder structure looks like : . ├── __init__.py ├── dags │ ├── __init__.py │ └── my_dag │ ├── __init__.py │ └── dag.py │ └── utils │ ├── __init__.py │ └── file.py │ …
Sébastien
  • 33
  • 4
3
votes
1 answer

AWS MWAA: 'Postgres not reachable' error when trying to run a local Apache Airflow environment on Docker

Following the AWS MWAA (Amazon Managed Workflows for Apache Airflow) installation tutorial for Windows using Docker in WSL (Windows Subsystem for Linux) 2 with Ubuntu distro, I encountered a problem when trying to run a local Apache Airflow…
Stan Lyons
  • 71
  • 5
3
votes
2 answers

Managing DBT profile file in MWAA

I would like to use DBT in MWAA Airflow enviroment. To achieve this I need to install DBT in the managed environment and from there run the dbt commands via the Airflow operators or CLI (BashOperator). My problem with solution is that I need store…
PeterRing
  • 1,767
  • 12
  • 20
3
votes
1 answer

AWS Managed Apache airflow (MWAA) or AWS Batch for simple batch job flow

I have simple workflow to design where there will be 4 batch job running one after another sequentially and each jobs is running in multi node master/slave architecture. My question is AWS Batch can manage simple workflow using job queue and can…
Ankit Gautam
  • 125
  • 1
  • 11
1
2
3
…
19 20