0

I'm using the Python APScheduler to run a bunch of recurring tasks, but for a couple tasks I now get errors such as:

WARNING:apscheduler.scheduler:Execution of job "getAndStoreExchangeRate (trigger: cron[minute='*'], next run at: 2014-11-06 18:48:00 CET)" skipped: maximum number of running instances reached (1)

In addition to that, the amount of used memory of this APscheduler instance rises suddenly after about ten days of running, from around 47MiB to 1200MiB, after which all sorts of processes stop because the machine is out of memory.

So to get to the root of the problem, I need to understand exactly which calls cause these warnings. As far as I understand, this error occurs because the previous call (a minute before this one) didn't end yet. The information in this error is rather vague though. the name of the function is given (getAndStoreExchangeRate), but I've got multiple files with a function named like this. So what I would like to know is:

  • in which file on which line is this function for which the warning occurs?
  • which arguments were given to this function?

Does anybody know how I can log this information somewhere? All tips are welcome!

[EDIT]

So I subclassed the BackgroundScheduler and overwrote the _process_jobs() method in which I changed the self._logger.warning in this snippet:

try:
    executor.submit_job(job, run_times)
except MaxInstancesReachedError:
    self._logger.warning(
        'Execution of job "%s" skipped: maximum number of running instances reached (%d)',
        job, job.max_instances
    )
except:
    self._logger.exception('Error submitting job "%s" to executor "%s"', job, job.executor)

to this:

'Execution of job "%s" skipped: maximum number of running instances reached (%d) =-= ARGS: ' + str(job.args) + ' - KWARGS: ' + str(job.kwargs),

This works in that it shows me the arguments given to the function, but I still don't know the most important thing: in which file and on which line this function for which the warning occurs is defined. Does anybody know how I could show that?

kramer65
  • 50,427
  • 120
  • 308
  • 488

1 Answers1

2

The easiest way would be to add an extra logging statement in the "except MaxInstancesReachedError:" block on apscheduler.schedulers.base module in the _process_jobs() method. There you have access to job.func which is the target function of the job.

I suggest you subclass your scheduler of choice and copy over the _process_jobs method code. Then modify the except MaxInstancesReachedError block:

try:
    executor.submit_job(job, run_times)
except MaxInstancesReachedError:
    self._logger.warning('Execution of job "%s" skipped: maximum number of running instances reached (%d)', job.func, job.max_instances)

EDIT:

The function's file name: job.func.__code__.co_filename

Line number: job.func.__code__.co_firstlineno

Alex Grönholm
  • 5,563
  • 29
  • 32
  • Thanks for the tip. Would you maybe have a tip on how I could do this? Do I need to subclass the BaseExecuter and override the `submit_job` method? Or do I need to fork the full code and mess around with that. You would greatly help me out with a little code example.. – kramer65 Dec 06 '14 at 08:49
  • Hi Alex, thanks a million for that. It got me a lot further. I'm again a bit stuck though; I've only got the `job` variable, but how do I actually get the info I want to log (the arguments supplied to this job and the file and line number on where the job function is defined)? Sorry to ask you again, but you would really make me so happy if I can solve this. – kramer65 Dec 09 '14 at 14:23
  • They are properties of the Job instance. Just look at the API documentation for reference. – Alex Grönholm Dec 11 '14 at 11:41
  • Hi Alex. Thanks again for the help. I'm almost there now. I edited my question to show how I subclassed the `_process_jobs()` method. The last thing I don't know is how I can find out in which file the called function is defined? I guess I would need the inspect module for that somehow, but I'm kinda lost in how I could do that. Any idea? (after that I promise I'll stop bothering you.. :) ) – kramer65 Dec 14 '14 at 13:23