Stack Overflow
Stack Overflow

Questions tagged [mwaa]

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed service for Apache Airflow that makes it easy for you to build and manage your workflows in the cloud.

you can easily combine data using any of Apache Airflow’s open source integrations.

use the same familiar Airflow platform as you do today to manage their workflows and now enjoy improved scalability, availability, and security without the operational burden of having to manage the underlying infrastructure

automatically scales capacity up to meet demand and back down to conserve resources and minimize costs

integrated with AWS security services to enable secure access

manages the provisioning and ongoing maintenance of Apache Airflow

https://docs.aws.amazon.com/mwaa/

291 questions
0
votes
1 answer

Airflow : aws-mwaa-local-runner :Unable to get SecretManager Connections from within docker

I am using https://github.com/aws/aws-mwaa-local-runner to test Airflow locally Issue: Unable to get connection details from…
Karthik
  • 441
  • 5
  • 17
0
votes
0 answers

Executing Php scripts in Airflow worker

I am a newbie to Airflow. I am working on a project , where i need to execute a php script in Airflow worker node.I am using S3 bucket to fetch the said scripts , but when i trigger them using bash command , I get php command not found error.Also…
khilari hemanshu
  • 1
  • 2
0
votes
1 answer

Timeout error invoke lambda function in airflow

I have a question regarding AWS lambda(seoul region) invokation from Airflow(Amazon Managed Workflows for Apache Airflow in tokyo region). The problem is when I invoke a Lambda function from Airflow, the Airflow UI says the function failed. The…
hyeonkimmm
  • 1
  • 1
0
votes
1 answer

Apache Manged Airflow EMR operator DAG is failing

I'm creating DAG in apache managed airflow workflows environment. DAG basiclly uses EMR operator , it creates EMR cluster, Run spark job defined in DAG and deletes the EMR cluster. I have taken the code from airflow website. This is the link for the…
dhanabalan Rangasamy
  • 319
  • 4
  • 18
0
votes
2 answers

Managed AWS Airflow with ECS Fargate DAG Jobs

Reading this article: https://aws.amazon.com/blogs/containers/running-airflow-on-aws-fargate/ It isn't clear to me; is it possible to run MWAA and execute jobs on ECS Fargate? Or to execute jobs on Ecstatics Fargate do you need to run then entire…
Myles McDonnell
  • 12,943
  • 17
  • 66
  • 116
0
votes
0 answers

Issue with Variable access in aws airflow

when I tried to access varaibles in aws managed airflow 2.0.1, got the below exception. psycopg2.errors.UndefinedColumn: column variable.description does not exist LINE 1: ....id AS variable_id, variable.key AS variable_key, variable.d... vars =…
John
  • 1,531
  • 6
  • 18
  • 30
0
votes
1 answer

AWS Airflow v2.0.2 doesn't show Postgres connection type

I just started migrating my Airflow v2.0.2 codebase to MWAA (AWS Airflow service). I added the following to the requirements.txt (and uploaded it to the S3 bucket intended for sync): apache-airflow-providers-postgres==2.0.0 But Postgres connection…
kee
  • 10,969
  • 24
  • 107
  • 168
0
votes
0 answers

error in importing pythin modules in mwaa dags

I am using MWAA. I have included pysftp in mwaa requirements.txt file as pysftp==0.2.9 In the scheduler log groups of MWAA I can even see that it is getting downloaded. Collecting pysftp==0.2.9 Downloading pysftp-0.2.9.tar.gz (25 kB) but when I…
Vibhav
  • 181
  • 5
  • 11
0
votes
1 answer

AWS MWAA: Some of the provided configurations do not have the expected format: scheduler, e.g: core.log_format

I am deploying Managed Airflow (MWAA) in AWS and getting the below error in the cloudformation. I am giving the log_format same as what mentioned in the airflow documentation but still CFN is giving the error. Resource handler returned message:…
Gagan
  • 1,775
  • 5
  • 31
  • 59
0
votes
1 answer

Postgres module not found in Airflow?

For some reason I'm having an issue importing from my airflow postgres packages. Any ideas? Error: Modules: Requirement.txt file with packages:
ZackR
  • 1
  • 1
0
votes
1 answer

Disable ParameterValidation in MWAA Airflow

In MWAA, I am using the following code to access the files in my S3 bucket. The S3 bucket is of the following form: aws s3 ls s3://example-bucket/incoming/driver-events/ingestDate=2021-05-26/ The above command works fine. Now I am attempting to get…
AnupamB
  • 31
  • 4
0
votes
1 answer

Apache Airflow - Slow to parse SQL queries on AWS MWAA

I'm trying to build a DAG on AWS MWAA, this DAG will export data from Postgres (RDS) to S3, but it's getting an issue once the MWAA tries to parse all queries from my task, in total it will export 385 tables, but the DAG gets stuck on running mode…
Doug P
  • 169
  • 1
  • 3
  • 13
0
votes
2 answers

AWS MWAA: Glue Crawler issue

I have manually provisioned a Glue Crawler and now am attempting to run it via Airflow (in AWS). Based on the docs from here, there seems to be plenty of ways to handle this objective compared to other tasks within the Glue environment. However, I'm…
nate
  • 440
  • 2
  • 8
  • 18
0
votes
2 answers

MWAA (aws managed airflow) failed to pause DAG when using CLI

I am trying to pause a DAG as described here but getting 'Dag id testDag not found' even though the DAG exists. The error message says that my 3rd party module 'not found' even though it works when I trigger the DAG. I saw this on the doc…
user3099614
  • 19
  • 1
  • 3
0
votes
0 answers

Downloaded S3 file to airflow tmp\test\ to process python scripts. First time job ran file. Second time it abended with template not found

I have requirement to download couple for files from s3 to aiflow tmp\test\ folder to process python script and load the file back to s3. Step1 : created the folder Step2 : download file 1 using bash script Step3 : download file 2 using bash…
rose1110
  • 49
  • 8
1 2 3
…
19
20