Send scheduled emails with pyramid_mailer and apscheduler

Question

I've tried getting this to work but there must be a better way, any input is welcome.

I'm trying to send scheduled emails in my python pyramid app using pyramid_mailer (settings stored in .ini file), and apscheduler to set the schedule.

I also use the SQLAlchemyJobStore so jobs can be restarted if the app restarts.

jobstores = {
    'default': SQLAlchemyJobStore(url='mysql://localhost/lgmim')
}
scheduler = BackgroundScheduler(jobstores=jobstores)

@view_config(route_name='start_email_schedule')
def start_email_schedule(request):
    # add the job and start the scheduler
    scheduler.add_job(send_scheduled_email, 'interval', [request], weeks=1)
    scheduler.start()

    return HTTPOk()

def send_scheduled_email(request):

    # compile message and recipients
    # send mail  
    send_mail(request, subject, recipients, message)

def send_mail(request, subject, recipients, body):

    mailer = request.registry['mailer']
    message = Message(subject=subject,
                  recipients=recipients,
                  body=body)

    mailer.send_immediately(message, fail_silently=False)

This is as far as I've gotten, now I'm getting an error, presumably because it can't pickle the request.

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

Using pyramid.threadlocal.get_current_registry().settings to get the mailer works the first time, but thereafter I get an error. I'm advised not to use it in any case.

What else can I do?

Answer 1

Generally, you cannot pickle request object as it contains references to things like open sockets and other liveful objects.

Some useful patterns here are that

• You pregenerate email id in the database and then pass id (int, UUID) over scheduler

• You generate template context (JSON dict) and then pass that over the scheduler and render the template inside a worker

• You do all database fetching and related inside a scheduler and don't pass any arguments

Specifically, the problem how to generate a faux request object inside a scheduler can be solved like this:

from pyramid import scripting
from pyramid.paster import bootstrap

def make_standalone_request():
    bootstrap_env = bootstrap("your-pyramid-config.ini")
    app = bootstrap_env["app"]
    pyramid_env = scripting.prepare(registry=bootstrap_env["registry"])
    request = pyramid_env["request"]

    # Note that request.url will be always dummy,
    # so if your email refers to site URL, you need to 
    # resolve request.route_url() calls before calling the scheduler
    # or read the URLs from settings

    return request

Some more inspiration can be found here (disclaimer: I am the author).