Why MongoDB does not recognize my user on Celery connection?

Question

I have a Python/Flask webapp with MongoDB database on DotCloud hosting. I would like to set up Celery on the Python service using the MongoDB service as the Celery queue.

Celery starts and it finds my celeryconfig.py file, which contains:

BROKER_BACKEND = "mongodb"
BROKER_HOST = "data.myapp.dotcloud.com"
BROKER_PORT = 8567
BROKER_USER = 'myuser'
BROKER_PASSWORD = 'mypass'
CELERY_IMPORTS = ('myapp.worker', )
CELERY_IGNORE_RESULT = True

Now I have a problem with DB authentification.

Here what I did to create the user:

~# mongo
MongoDB shell version: 1.8.1
connecting to: test
-> use celery
switched to db celery
-> db.getSisterDB("admin").auth("root", "root_password_for_mongodb_service");
1
-> db.addUser("myuser", "mypass");

Here are the Celery logs (on the python service):

-------------- celery@myapp-www v2.3.1
---- **** -----
--- * ***  * -- [Configuration]
-- * - **** ---   . broker:      mongodb://myuser@data.myapp.dotcloud.com:8567/
- ** ----------   . loader:      celery.loaders.default.Loader
- ** ----------   . logfile:     [stderr]@WARNING
- ** ----------   . concurrency: 4
- ** ----------   . events:      OFF
- *** --- * ---   . beat:        OFF
-- ******* ----
--- ***** ----- [Queues]
 --------------   . celery:      exchange:celery (direct) binding:celery
[2011-08-26 14:18:10,453: WARNING/MainProcess] celery@myapp-www has started.
[2011-08-26 14:18:12,873: WARNING/MainProcess] Traceback (most recent call last):
[2011-08-26 14:18:12,874: WARNING/MainProcess] File "/home/dotcloud/env/bin/celeryd", line 8, in <module>
[2011-08-26 14:18:12,874: WARNING/MainProcess] load_entry_point('celery==2.3.1', 'console_scripts', 'celeryd')()
[2011-08-26 14:18:12,875: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/bin/celeryd.py", line 187, in main
[2011-08-26 14:18:12,875: WARNING/MainProcess] worker.execute_from_commandline()
[2011-08-26 14:18:12,876: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/bin/base.py", line 72, in execute_from_commandline
[2011-08-26 14:18:12,877: WARNING/MainProcess] return self.handle_argv(prog_name, argv[1:])
[2011-08-26 14:18:12,877: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/bin/base.py", line 100, in handle_argv
[2011-08-26 14:18:12,878: WARNING/MainProcess] return self.run(*args, **vars(options))
[2011-08-26 14:18:12,878: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/bin/celeryd.py", line 96, in run
[2011-08-26 14:18:12,878: WARNING/MainProcess] return self.app.Worker(**kwargs).run()
[2011-08-26 14:18:12,879: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/apps/worker.py", line 140, in run
[2011-08-26 14:18:12,879: WARNING/MainProcess] self.run_worker()
[2011-08-26 14:18:12,880: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/apps/worker.py", line 240, in run_worker
[2011-08-26 14:18:12,880: WARNING/MainProcess] worker.start()
[2011-08-26 14:18:12,880: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/worker/__init__.py", line 252, in start
[2011-08-26 14:18:12,881: WARNING/MainProcess] blocking(component.start)
[2011-08-26 14:18:12,881: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/syn.py", line 14, in blocking
[2011-08-26 14:18:12,882: WARNING/MainProcess] return __sync_current(fun, *args, **kwargs)
[2011-08-26 14:18:12,882: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/syn.py", line 30, in __blocking__
[2011-08-26 14:18:12,882: WARNING/MainProcess] return fun(*args, **kwargs)
[2011-08-26 14:18:12,883: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/worker/consumer.py", line 302, in start
[2011-08-26 14:18:12,883: WARNING/MainProcess] self.reset_connection()
[2011-08-26 14:18:12,883: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/worker/consumer.py", line 555, in reset_connection
[2011-08-26 14:18:12,884: WARNING/MainProcess] on_decode_error=self.on_decode_error)
[2011-08-26 14:18:12,884: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/celery/app/amqp.py", line 331, in get_task_consumer
[2011-08-26 14:18:12,885: WARNING/MainProcess] **kwargs)
[2011-08-26 14:18:12,885: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/compat.py", line 228, in __init__
[2011-08-26 14:18:12,885: WARNING/MainProcess] super(ConsumerSet, self).__init__(self.backend, queues, **kwargs)
[2011-08-26 14:18:12,886: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/messaging.py", line 242, in __init__
[2011-08-26 14:18:12,886: WARNING/MainProcess] self.declare()
[2011-08-26 14:18:12,887: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/messaging.py", line 252, in declare
[2011-08-26 14:18:12,887: WARNING/MainProcess] queue.declare()
[2011-08-26 14:18:12,888: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/entity.py", line 364, in declare
[2011-08-26 14:18:12,888: WARNING/MainProcess] self.name and self.queue_declare(nowait, passive=False),
[2011-08-26 14:18:12,888: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/entity.py", line 382, in queue_declare
[2011-08-26 14:18:12,889: WARNING/MainProcess] nowait=nowait)
[2011-08-26 14:18:12,889: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/syn.py", line 14, in blocking
[2011-08-26 14:18:12,889: WARNING/MainProcess] return __sync_current(fun, *args, **kwargs)
[2011-08-26 14:18:12,890: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/syn.py", line 30, in __blocking__
[2011-08-26 14:18:12,890: WARNING/MainProcess] return fun(*args, **kwargs)
[2011-08-26 14:18:12,890: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/transport/virtual/__init__.py", line 357, in queue_declare
[2011-08-26 14:18:12,891: WARNING/MainProcess] return queue, self._size(queue), 0
[2011-08-26 14:18:12,891: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/transport/mongodb.py", line 45, in _size
[2011-08-26 14:18:12,892: WARNING/MainProcess] return self.client.count()
[2011-08-26 14:18:12,892: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/transport/mongodb.py", line 81, in client
[2011-08-26 14:18:12,893: WARNING/MainProcess] self._client = self._open()
[2011-08-26 14:18:12,893: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/kombu/transport/mongodb.py", line 75, in _open
[2011-08-26 14:18:12,893: WARNING/MainProcess] col.ensure_index([("queue", 1)])
[2011-08-26 14:18:12,894: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/pymongo/collection.py", line 724, in ensure_index
[2011-08-26 14:18:12,894: WARNING/MainProcess] ttl, **kwargs)
[2011-08-26 14:18:12,895: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/pymongo/collection.py", line 646, in create_index
[2011-08-26 14:18:12,895: WARNING/MainProcess] safe=True)
[2011-08-26 14:18:12,895: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/pymongo/collection.py", line 275, in insert
[2011-08-26 14:18:12,896: WARNING/MainProcess] check_keys, safe, kwargs), safe)
[2011-08-26 14:18:12,897: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/pymongo/connection.py", line 770, in _send_message
[2011-08-26 14:18:12,897: WARNING/MainProcess] return self.__check_response_to_last_error(response)
[2011-08-26 14:18:12,898: WARNING/MainProcess] File "/home/dotcloud/env/lib/python2.6/site-packages/pymongo/connection.py", line 722, in __check_response_to_last_error
[2011-08-26 14:18:12,898: WARNING/MainProcess] raise OperationFailure(error["err"])
[2011-08-26 14:18:12,899: WARNING/MainProcess] pymongo.errors
[2011-08-26 14:18:12,899: WARNING/MainProcess] .
[2011-08-26 14:18:12,899: WARNING/MainProcess] OperationFailure
[2011-08-26 14:18:12,900: WARNING/MainProcess] :
[2011-08-26 14:18:12,900: WARNING/MainProcess] unauthorized

Here are the MongoDB logs (dotcloud logs myapp.data):

Fri Aug 26 14:19:43 [initandlisten] connection accepted from 10.68.47.216:43499 #6053
Fri Aug 26 14:19:43 [initandlisten] connection accepted from 10.68.47.216:43500 #6054
Fri Aug 26 14:19:43 [conn6054] auth: couldn't find user myuser, kombu_default.system.users
Fri Aug 26 14:19:43 [conn6054] query kombu_default.$cmd ntoreturn:1 command: { authenticate: 1, nonce: "d7feebcf0bdbb839", user: "myuser", key: "ad0b6da89f5bae34c70b34e2dac5b678" } reslen:76 256ms
Fri Aug 26 14:19:45 [conn6053] end connection 10.68.47.216:43499
Fri Aug 26 14:19:45 [conn6054] end connection 10.68.47.216:43500
Fri Aug 26 14:19:47 [initandlisten] connection accepted from 10.68.47.216:43527 #6055
Fri Aug 26 14:19:47 [initandlisten] connection accepted from 10.68.47.216:43530 #6056
Fri Aug 26 14:19:47 [conn6056] auth: couldn't find user myuser, kombu_default.system.users
Fri Aug 26 14:19:49 [conn6055] end connection 10.68.47.216:43527
Fri Aug 26 14:19:49 [conn6056] end connection 10.68.47.216:43530

According to you, where am I wrong?

Regards,

MV

Can you connect to the celery db from the command line? Try connecting with the pymongo driver to `mongodb://myuser:mypass@mongo data.myapp.dotcloud.com:8567/celery` and ensure that works first. — Gates VP, Aug 26 '11 at 19:23

score 2 · Accepted Answer · answered Aug 27 '11 at 16:06

2

Finally I learnt that "kombu_default" (in the MongoDB logs) was the database used for message queue by Celery.

So, to get Celery + MongoDB working, I only add the parameter BROKER_VHOST = "celery" to the celeryconfig.py file.

answered Aug 27 '11 at 16:06

Marc de Verdelhan

2,501
3
21
40

1

By the way, you should really consider using RabbitMQ instead of MongoDB as the broker for Celery. Unless you specifically want to be able to tweak the Celery jobs in MongoDB. If that's just about the total number of services allowed on DotCloud, their support can probably tweak up your limit to accommodate for that. – jpetazzo Sep 01 '11 at 00:42
1

@jpetazzo - I too have seen this admonishment just about everywhere celery and mongo are mentioned in the same sentence - I'm really curious as to why? – DeaconDesperado Mar 05 '12 at 15:49
1

RabbitMQ is a "real" message queue. MongoDB is a key/value database. You can of course use it to pass messages around (just like any SQL DB), but it's (IMHO!) a weird usage. Last time I checked, MongoDB didn't have support for asynchronous notifications; meaning that clients have to poll at regular intervals to check for new data. Also, MongoDB isn't lightweight. I think it has changed recently, but when I wrote my original comment, MongoDB was creating 2 GB files (mmap, then write in them) when you start it. A bit overkill when all you want is a queue with a few hundreds of messages, I think! – jpetazzo Mar 17 '12 at 16:37
... But for simplicity's sake, today I would change my mind and recommend people to use the Redis back-end. I think it's easier to install than MongoDB or RabbitMQ, it's definitely very light, and it has proper PUB/SUB support as well as proper queues with BLPOP. – jpetazzo Mar 17 '12 at 16:38

Why MongoDB does not recognize my user on Celery connection?

1 Answers1