I am attempting to configure an Airflow server on EC2 in AWS. I have setup the database to point to MySQL instance in RDS. I wanted to set it up using AWS SQS instead of RabbitMQ, although this is where I'm stuck.
I have it configured for the CeleryExecutor
My airflow.cfg entry for broker_url is the following format
broker_url = sqs://[AWS KEY]:[AWS SECRET]@[SQS URL]
and I've attempted
broker_url = sqs://[AWS KEY]:[AWS SECRET]@
And I get the following response
[ec2-user@ip-10-0-0-196 airflow]$ airflow worker
[2017-06-27 00:03:38,611] {__init__.py:57} INFO - Using executor CeleryExecutor
-------------- celery@ip-10-0-0-196 v4.0.2 (latentcall)
---- **** -----
--- * *** * -- Linux-4.9.27-14.33.amzn1.x86_64-x86_64-with-glibc2.2.5 2017-06-27 00:03:38
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: airflow.executors.celery_executor:0x7fd07c867f50
- ** ---------- .> transport: sqs://[AWS KEY]:**@localhost//
- ** ---------- .> results: mysql://[RDS DB USERNAME]:**@[RDS URL]:3306/airflowdb
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> default exchange=default(direct) key=default
[2017-06-27 00:03:39,270] {__init__.py:57} INFO - Using executor CeleryExecutor
Starting flask
[2017-06-27 00:03:39,545] {_internal.py:87} INFO - * Running on http://0.0.0.0:8793/ (Press CTRL+C to quit)
[2017-06-27 00:03:40,671: CRITICAL/MainProcess] Unrecoverable error: TypeError(u'Cannot mix str and non-str arguments
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 370, in start
return self.obj.start()
File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
blueprint.start(self)
File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 594, in start
c.loop(*c.loop_args())
File "/usr/local/lib/python2.7/site-packages/celery/worker/loops.py", line 88, in asynloop
next(loop)
File "/usr/local/lib/python2.7/site-packages/kombu/async/hub.py", line 282, in create_loop
item()
File "/usr/local/lib/python2.7/site-packages/vine/promises.py", line 139, in __call__
return self.throw()
File "/usr/local/lib/python2.7/site-packages/vine/promises.py", line 136, in __call__
retval = fun(*final_args, **final_kwargs)
File "/usr/local/lib/python2.7/site-packages/kombu/transport/SQS.py", line 290, in _schedule_queue
queue, callback=promise(self._loop1, (queue,)),
File "/usr/local/lib/python2.7/site-packages/kombu/transport/SQS.py", line 306, in _get_bulk_async
return self._get_async(queue, maxcount, callback=callback)
File "/usr/local/lib/python2.7/site-packages/kombu/transport/SQS.py", line 316, in _get_async
callback=transform(self._on_messages_ready, callback, q, queue),
File "/usr/local/lib/python2.7/site-packages/kombu/transport/SQS.py", line 336, in _get_from_sqs
callback=callback,
File "/usr/local/lib/python2.7/site-packages/kombu/async/aws/sqs/connection.py", line 82, in receive_message
queue.id, callback=callback,
File "/usr/local/lib/python2.7/site-packages/kombu/async/aws/connection.py", line 239, in get_list
self._on_list_ready, callback, parent or self, markers,
File "/usr/local/lib/python2.7/site-packages/kombu/async/aws/connection.py", line 232, in make_request
return self._mexe(request, callback=callback)
File "/usr/local/lib/python2.7/site-packages/kombu/async/aws/connection.py", line 198, in _mexe
conn.getresponse(callback=callback)
File "/usr/local/lib/python2.7/site-packages/kombu/async/aws/connection.py", line 127, in getresponse
request = self.getrequest()
File "/usr/local/lib/python2.7/site-packages/kombu/async/aws/connection.py", line 120, in getrequest
url = urlunsplit((scheme, host, self.path, '', ''))
File "/usr/local/lib/python2.7/site-packages/future/backports/urllib/parse.py", line 399, in urlunsplit
_coerce_args(*components))
File "/usr/local/lib/python2.7/site-packages/future/backports/urllib/parse.py", line 115, in _coerce_args
raise TypeError("Cannot mix str and non-str arguments")
TypeError: Cannot mix str and non-str arguments
Both the Key and Secret do not contain any special characters.
Does anyone have a successful Airflow setup using SQS that they could share their airflow.cfg?
Thanks Anant
