Stack Overflow
Stack Overflow

Questions tagged [mwaa]

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed service for Apache Airflow that makes it easy for you to build and manage your workflows in the cloud.

you can easily combine data using any of Apache Airflow’s open source integrations.

use the same familiar Airflow platform as you do today to manage their workflows and now enjoy improved scalability, availability, and security without the operational burden of having to manage the underlying infrastructure

automatically scales capacity up to meet demand and back down to conserve resources and minimize costs

integrated with AWS security services to enable secure access

manages the provisioning and ongoing maintenance of Apache Airflow

https://docs.aws.amazon.com/mwaa/

291 questions
0
votes
1 answer

How to set an environment variable in AWS MWAA

I'm trying to set the GOOGLE_APPLICATION_CREDENTIALS environment variable in MWAA to authenticate Google Cloud, but I can't figure out how. In GCP Composer I can just use the console and add an environment variable, is there nothing like that in AWS…
pshaw20
  • 103
  • 2
  • 7
0
votes
2 answers

How to run some batch script on all MWAA Airflow worker nodes?

Is there a way to run a specific batch script on every worker node that MWAA spins up? Is there any feature in airflow that can do this?
rawwar
  • 4,834
  • 9
  • 32
  • 57
0
votes
0 answers

How to upgrade SQLite in AWS MWAA Airflow?

I am using yfinance library to get some values like market_cap. Code:- import yfinance as yf com = yf.Ticker('1140.SR') print(com.fast_info['market_cap']) I've updated the library to latest version i.e. 0.2.9 locally and on AWS MWAA Apache…
Jasmohan
  • 237
  • 3
  • 5
0
votes
1 answer

Where do I put a custom config (.cfg) file in AWS MWAA Airflow?

I have a config file, dev.cfg, that looks like this: [S3] bucket_name = my-bucket I need this in my code to do S3 things. I do not have the access (nor will I be given it) to modify the environmental or config variables in the AWS console. My only…
offwhitelotus
  • 1,049
  • 9
  • 15
0
votes
1 answer

How to generate templated Airflow DAGs using Jinja

I'm bit new to Airflow and was exploring creation of multiple DAGs that have more or less same code from a template instead of creating them as individual DAGs which introduces maintenance overhead. I found this article on medium and it works well…
ACL
  • 409
  • 3
  • 4
0
votes
0 answers

MWAA & SlackWebHookOperator - SSLError: EOF occurred in violation of protocol

Background One use case I have with MWAA is sending out Slack notifications by the use of Slack Incoming Webhook. To do so, I have installed the Slack Provider package so that I could create a connector and make use of the operator called…
Caio Lauro
  • 1
  • 1
0
votes
2 answers

How to get region-name from DAG in MWAA

I would like to know how to get a region name like ap-northeast-1 from DAG in MWAA. Is it possible to get it? We can get region name from config file. However, we would like to refer it without config file.
YOYO
  • 25
  • 3
0
votes
1 answer

MWAA trigger dag issue

I am using MWAA, and when I try to trigger a dag, sometimes I receive this error. I am not sure what to do about this error. def should_retry(response) -> bool: """ If `true`, then retry the Airflow job. Else, fail. """ …
Nelly L
  • 31
  • 2
0
votes
0 answers

Multiple jobs getting triggered instead of only one job in MWAA

We are facing an issue in Managed Appflow Apache Airflow(MWAA) service, we created 10 different DAGs with almost same DAG operators (AWS Glue job) but with different parameters to same job. We face an issue that operators in multiple DAGs are…
Sridhar
  • 11
  • 2
0
votes
2 answers

Is there a way to import variables and connections to MWAA?

I have a MWAA environment and I have to create another one by Terraform. The environment creation is not an issue, but the 'metadata' of my old enviroment. I want to import all variables and connections, programatically, but I haven't figured out so…
Luis Felipe
  • 148
  • 9
0
votes
2 answers

Trigger a dag in Amazon Managed Workflows for Apache Airflow (MWAA) as a part CI/CD

Wondering if there is any way (blueprint) to trigger an airflow dag in MWAA on the merge of a pull request (preferably via github actions)? Thanks!
AlizaminJ
  • 95
  • 1
  • 5
0
votes
0 answers

Airflow v2.2.5 on MWAA v2.0.2 is queuing but not running DAG tasks

I'm running Airflow v2.2.5 on MWAA v2.0.2 locally on Docker v20.10.12. I checked the basic debugging checklist for such cases (see here) but I don't see any issue with scheduler or webserver not running, improper start date or schedule, etc. Here's…
Myklebost
  • 59
  • 8
0
votes
0 answers

Apache airflow issue while trying to update new script from S3

I've a very specific issue and need your help, appreciate any help in this regard. I'm using the MWAA (AWS) airflow and the DAG details are picked up from S3. Recently we found that one of the DAGs was not configured properly, hence we updated the…
TJ-AWS
  • 83
  • 8
0
votes
1 answer

Get Airflow Dag Concurrent Runs in MWAA

Can we get the airflow dag run count( concurrency) , which runs in MWAA? I set the concurrency on Dag Level. But I would like to get dag concurrency, and if it is below a specific limit would like to trigger the dag. Since the dag is running in MWAA…
KK2486
  • 353
  • 2
  • 3
  • 13
0
votes
0 answers

How to PIP install package in a GitHub repo on MWAA

I have made a python package and hosted it on GitHub,it gets installed fine in my local machine using pip install git+url , but when I put it in the requirements.txt for AWS MWAA it's not getting installed. tried putting the git+url in…
Adhyayan Srijan
  • 1
1 2 3
…
19 20