3

I need help in passing the arguments (conf params) to mwaa (airflow) from lambda. Lmabda is used to trigger the dag on sqs event.

The dag runs fine without command line params.

import boto3
import http.client
import base64
import ast

mwaa_env_name = 'dflow_dev_2'
dag_name = 'tpda_test'
mwaa_cli_command = 'dags trigger'


client = boto3.client('mwaa')
def lambda_handler(event, context):
    # get web token
    mwaa_cli_token = client.create_cli_token(
        Name=mwaa_env_name
    )
    
    conn = http.client.HTTPSConnection(mwaa_cli_token['WebServerHostname'])
    payload = "dags trigger " + dag_name + "--conf '{'name':'v111'}' "
    headers = {
      'Authorization': 'Bearer ' + mwaa_cli_token['CliToken'],
      'Content-Type': 'text/plain'
    }
    conn.request("POST", "/aws_mwaa/cli", payload, headers)
    res = conn.getresponse()
    data = res.read()
    dict_str = data.decode("UTF-8")
    mydata = ast.literal_eval(dict_str)
    return base64.b64decode(mydata['stdout'])
user3858193
  • 1,320
  • 5
  • 18
  • 50
  • Were you able to resolve this? I am facing the exact same issue. Without conf, the DAG is triggered as expected. However, with conf, the DAG is not getting triggered at all. I am invoking the DAG in the same way as shown in your example. Any luck finding the solution? – Siddharth Gosalia May 10 '22 at 05:11
  • I know it's too late but the same code is working fine in my case too. It's just that the way of payload and conf should be changed as mentioned in the answer below. @SiddharthGosalia – RushHour Mar 27 '23 at 18:48

1 Answers1

1

There may be an issue with payload. The code example below adds a space between the DAG name and --conf and changes the quotations in the JSON string to double quotations.

conf = "{\"" + "name" + "\":\"" + "v111" + "\"}"
payload = "dags trigger " + dag_name + " --conf '{}'".format(conf)

Notice the different values for payload before and after.

Before:
dags trigger tpda_test--conf '{'name':'v111'}' 

After:
dags trigger tpda_test --conf '{"name":"v111"}'

Reference: Add a configuration when triggering a DAG (AWS)

Andrew Nguonly
  • 2,258
  • 1
  • 17
  • 23