Flask mail giving Pickling errors with celery

Question

I trying to use Celery (and rabbitmq) to send emails asynchronously with Flask mail. Initially I had an issue with render_template from flask breaking celery - Flask-Mail breaks Celery (The celery task would still execute successfully but no emails were being sent). While I was trying to fix that issue (which is still not fixed!) - I stumbled upon another problem. This pickling error which is due to a thread lock. I noticed that the problem started when I changed the way I called the celery task (from delay to apply_async). Since then I tried reverting my changes but I still can't get rid of the error. Any help regarding any one of the issues will be highly appreciated.

The traceback:

File "/Users/.../python2.7/site-packages/celery/app/amqp.py",          line 250, in publish_task
    **kwargs
File "/Users/.../lib/python2.7/site-packages/kombu/messaging.py", line 157, in publish
compression, headers)
File "/Users/.../lib/python2.7/site-packages/kombu/messaging.py", line 233, in _prepare
    body) = encode(body, serializer=serializer)
File "/Users/.../lib/python2.7/site-packages/kombu/serialization.py", line 170, in encode
    payload = encoder(data)
File "/Users/.../lib/python2.7/site-packages/kombu/serialization.py", line 356, in dumps
    return dumper(obj, protocol=pickle_protocol)
PicklingError: Can't pickle <type 'thread.lock'>: attribute lookup thread.lock failed

tasks.py

from __future__ import absolute_import
from flask import render_template
from flask.ext.mail import Message
from celery import Celery

celery = Celery('tasks', 
            broker = 'amqp://tester:testing@localhost:5672/test_host')

@celery.task(name = "send_async_email")
def send_auth_email(app, nickname, email):
    with app.test_request_context("/"):
        recipients = []
        recipients.append(email)
        subject = render_template("subject.txt")
        msg = Message(subject, recipients = recipients)
        msg.html = render_template("test.html", name = nickname)
        app.mail.send(msg)

In the test case I just call:

send_auth_email.delay(test_app, nick, email)

FYI: The API works perfectly fine if I don't use celery (i.e. synchronously). Thanks in advance!

alko · Answer 1 · 2013-10-24T14:09:45.167

1

When you invoke send_auth_email.delay(test_app, nick, email) all function arguments are being sent to task Queue. To do so, Celery pickles them.

Short answer test_app, being flask application, uses some magic and cannot be pickled. See docs for details on what can be pickled, and what not.

One solution is to pass all necessary arguments (in your case it seems that this is only name) to re-instantiate test_app in send_auth_email.

edited Oct 24 '13 at 14:09

answered Oct 24 '13 at 13:47

alko

46,136
12
94
102

It is an instance of the flask application. FYI - I don't have a celeryconfig.py file and a result backend. Should that be an issue? – user2216194 Oct 24 '13 at 14:00
no, this will lead further errors, this one clearly do not depends on that. updated answer according to your reply. – alko Oct 24 '13 at 14:10

Flask mail giving Pickling errors with celery

1 Answers1