Stack Overflow
Stack Overflow

Questions tagged [mwaa]

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed service for Apache Airflow that makes it easy for you to build and manage your workflows in the cloud.

you can easily combine data using any of Apache Airflow’s open source integrations.

use the same familiar Airflow platform as you do today to manage their workflows and now enjoy improved scalability, availability, and security without the operational burden of having to manage the underlying infrastructure

automatically scales capacity up to meet demand and back down to conserve resources and minimize costs

integrated with AWS security services to enable secure access

manages the provisioning and ongoing maintenance of Apache Airflow

https://docs.aws.amazon.com/mwaa/

291 questions
0
votes
1 answer

Where is a sensible place to put kube_config.yaml files on MWAA?

The example code in the MWAA docs for connecting MWAA to EKS has the following: #use a kube_config stored in s3 dags folder for now kube_config_path = '/usr/local/airflow/dags/kube_config.yaml' This doesn't make me think that putting the…
LondonRob
  • 73,083
  • 37
  • 144
  • 201
0
votes
1 answer

Retrive a Xcomm value and pass it to spark _steps in EMR operator, Airflow

I am trying to retrieve value from a python operator and pass it to "EMR create job" and "add steps operator". How do I pass this value in spark_steps, I have used task_instance to access it but it didn't work, can someone please tell me how to…
Xi12
  • 939
  • 2
  • 14
  • 27
0
votes
0 answers

RunJobFlow operation: An instance group must have at least one instance

I am writing code to run a psyaprk job using AIrflow 2.0.2,and I only need to start the master nodes, no core nodes My JOB_FLOW_OVERRIDES is throwing an error saying: An instance group must have at least one instance. I already have InstanceCount=…
Xi12
  • 939
  • 2
  • 14
  • 27
0
votes
0 answers

MWAA Airflow UI: Broken plugin[No module]

I have created Airflow environment in MWAA. I have mentioned all the required modules in requirements.txt but getting 'Broken plugin: No module found' error in MWAA Airflow UI although Scheduler/worker logs show modules are installed successfully…
Rjkant
  • 1
0
votes
1 answer

Polling multiple SQS messages using Airflow SQSSensor

I am using this SQSSensoe settings to poll messages fetch_sqs_message = SQSSensor( task_id="...", sqs_queue="...", aws_conn_id="aws_default", max_messages=10, wait_time_seconds=30, …
Kei
  • 611
  • 2
  • 11
  • 24
0
votes
1 answer

How to hide aws_key_id and aws_secret_key under Rendered Template in AWS MWAA

I am using AWS Managed Airflow (MWAA) with Airflow version 2.0.2. To setup External Tables in Snowflake I am using IAM User and I need to pass aws_key_id and aws_secret_key in Create Stage statements. I have templated my Create Stage statements and…
Mohan Hegde
  • 11
  • 4
0
votes
0 answers

Airflow - What do I do when I have a variable amount of Work that needs to be handled by a DAG?

I have a sensor task that listens to files being created in S3. After a poke I may have 3 files, after another poke I might have another 5 files. I want to create a DAG (or multiple dags) that listen to work request, and creates others tasks or DAGs…
zeh
  • 1,197
  • 2
  • 14
  • 29
0
votes
1 answer

MWAA - Impact of configuring a requirements.txt file

I am working with Amazon Managed Workflows for Apache Airflow (MWAA). We have some DAGs running successfully in the instance. They use the bash operator and the python operator. We do not currently have a requirments.txt file specified for the…
MattG
  • 5,589
  • 5
  • 36
  • 52
0
votes
1 answer

MWAA - Airflow Simple Python Operator Usage for code organised in multiple files using local imports

I am playing with Amazon Managed Workflows for Apache Airflow (MWAA) for the first time, so I could be missing some basics. If I had a python application, which I had organised/spread across 2 files/scripts so it looked like…
MattG
  • 5,589
  • 5
  • 36
  • 52
0
votes
2 answers

aws mwaa is not creating cloud watch log streams

It logs the below error in the Task. This same task has no issues in the dev environment. This issue started in prod environment sometime after mid Feb 2022. Noticed it now. Nothing has changed. Could not read remote logs from log_group:…
joss
  • 695
  • 1
  • 5
  • 16
0
votes
0 answers

Extract connections in airflow and use in boto3

I am. trying to extract connections from airflow as key,pass = BaseHook.get_connection('aws_default') print(conn.get_extra()) and use it in my boto3 connection as: account_id =…
Xi12
  • 939
  • 2
  • 14
  • 27
0
votes
1 answer

why does my EMR terminate after last step, when started through Airflow?

I am using the code below to run EM, copy and execute a shell script. cluster_creator == Starting a cluster through EMR step_adder == Executing Shell scripts step_checker == Checking if steps are done. My question is, why does EMR terminate after…
Xi12
  • 939
  • 2
  • 14
  • 27
0
votes
2 answers

I have created an MWAA Cluster using terraform. I am trying to import variables using curl command. But it fails with an 403 Exception

I have created an MWAA Cluster using terraform. I am trying to import variables using the curl command. But it fails with a 403 Exception. Airflow Version: v1.10.12 Exception: You don't have the permission to access the requested resource. It is…
Pavan
  • 31
  • 6
0
votes
1 answer

Airflow 2.2 TriggerDagRunOperator wait_for_completion behavior

I have a question about the TriggerDagRunOperator , specifically the wait_for_completion parameter. Before moving to Airflow 2.2, we used this operator to trigger another DAG and a ExternalTaskSensor to wait for its completion. In Airflow 2.2, there…
snir.isl
  • 155
  • 1
  • 12
0
votes
2 answers

How to get reason for failure using slack in airflow2.0

How to get the reason for the failure of an operator, without going into logs. As I want to post the reason as a notification through slack? Thanks, Xi
Xi12
  • 939
  • 2
  • 14
  • 27
1 2 3
…
19 20