0

The code is here.

I wrote an extension of the celery.schedules.schedule interface, and I can't figure out why it's getting instantiated with nothing set in the extra values I created.

When I instantiate them before passing to app.conf.CELERYBEAT_SCHEDULE they're correct. But all the ones that celery beat instantiates are incorrect.

I asked in #celery IRC chan and the only response I got was about lazy mode, but that's for celery.beat.Scheduler, not celery.schedules.schedule, so if it's relevant, I don't understand how. Do I have to extend that too, just so that it instantiates the schedules correctly?

I've tried digging into the celery code w/the debugger to figure out where these schedules are getting instantiated and I can't find it. I can see when they come back from Unpickler they are wrong, but I can't find where they get created or where they get pickled.

Charles Thomas
  • 3
  • 1

2 Answers2

0

After a lot of time in the Python debugger, I narrowed the problem down to celery.beat.PersistentScheduler.sync() and/or shelve.sync() (which is called by the former).

When the shelve is synced, the values are getting lost. I don't know why, but I'm pretty sure it's a bug in either Celery or Shelve.

In any case, I wrote a workaround.

Charles Thomas
  • 3
  • 1
0

celery.schedules.schedule has a __reduce__ method that defines how to serialize and reconstruct the object using pickle:

https://github.com/celery/celery/blob/master/celery/schedules.py#L150-L151

When pickle serializes the object it will call:

fun, args = obj.__reduce__()

and when it reconstructs the object it will do:

obj = fun(*args)

So if you've added new state to your custom schedule subclass, passed as arguments to __init__, then you will also have to define a __reduce__ method that takes these new arguments into account:

class myschedule(schedule):

    def __init__(self, run_every=None, relative=False, nowfun=None,
                 odds=None, max_run_every=None, **kwargs):
        super(myschedule, self).__init__(
            run_every, relative, nowfun, **kwargs)
        self.odds = odds
        self.max_run_every = max_run_every

    def __reduce__(self):
        return self.__class__, (
            self.run_every, self.relative, self.nowfun,
            self.odds, self.max_run_every)
asksol
  • 19,129
  • 5
  • 61
  • 68