0

I have a cron job that runs a django management command that clears my mail queue. It turns out it wasn't sending, and several thousand customers had not received emails.

Any ideas what's going on here?

I have this line in my crontab (and has been the same for the past 3+ years.)

* * * * * /srv/grove_project/bin/cron.bash send_mail

When I run it by the same user manually it works. 

/srv/grove_project/bin/cron.bash send_mail

In /var/log/syslog there is 

Sep 20 10:10:01 ~ CMD (/srv/grove_project/bin/cron.bash process_fulfillment_postback > /dev/null 2>&1)
Sep 20 10:10:01 ~ CMD (/srv/grove_project/bin/cron.bash send_mail )
Sep 20 10:10:01 ~ CMD (/srv/grove_project/bin/cron.bash monitor > /dev/null 2>&1)

The other commands are working fine.

What gives?

Update

More information: the output of the command is:

/srv/grove_project/env/lib/python2.6/site-packages/django/conf/__init__.py:75: DeprecationWarning: The ADMIN_MEDIA_PREFIX setting has been removed; use STATIC_URL instead.
  "use STATIC_URL instead.", DeprecationWarning)
INFO 2012-09-20 12:34:02,210 root 31267 ------------------------------------------------------------------------

Here's the function spitting that out:

    logging.info("-" *  72) #<--- this line printed
    # if PAUSE_SEND is turned on don't do anything.
    if not PAUSE_SEND: 
        send_all()
    else: 
        logging.info("sending is paused, quitting.")

I'm not seeing the "sending is paused" message, so looking into the send_all function

lock = FileLock("send_mail")
logging.debug("acquiring lock...") #<-- I don't see this line

Since I don't see the acquiring lock message, it must be silently hanging on FileLock only for crontab?

Yuji Tomita
  • 463
  • 1
  • 3
  • 15

2 Answers2

1

Append this > /tmp/send_mail.log 2>&1 to the end of your cron, then wait for it fail again, and take a look at the send_mail.log to see what happens.

What is the difference in environment between calling the script myself vs crontab?

Cron runs under a very minimal environment.

You can redirect your current env to a file:

env > ~/env.full

then run your job with this environment:

* * * * * env - $(/bin/cat /home/yuji/env.full) /srv/grove_project/bin/cron.bash send_mail > /tmp/send_mail.log 2>&1

quanta
  • 51,413
  • 19
  • 159
  • 217
  • I keep getting one log line out of the python function. What is the difference in environment between calling the script myself vs crontab that might account for differing behavior in the python code? I can't really account for this in the mailer code. Especially confusing is the fact that it suddenly broke. – Yuji Tomita Sep 20 '12 at 17:31
  • I'm getting env: 60418: No such file or directory. Tried /usr/bin/env too? – Yuji Tomita Sep 20 '12 at 17:59
  • Which operating system are you running? – quanta Sep 20 '12 at 18:02
  • I'm using Ubuntu 10.4. It sounds python related.. the environment is loading. Django is loading. Other commands using django environment, models, etc., are working via cron. Hmmm – Yuji Tomita Sep 20 '12 at 18:09
  • Change it to `printenv` and try again. – quanta Sep 20 '12 at 18:16
  • I get no output. Do you think it's an environment issue? I don't think so because the python environment successfully loads in other places. I'm going to see what happens if I try to direct call the python function from another currently working cronjob (instead of relying on manage.py) – Yuji Tomita Sep 20 '12 at 19:01
1

Send the output an error to a logfile. Instead of this:

* * * * * /srv/grove_project/bin/cron.bash send_mail

Do this:

* * * * * /srv/grove_project/bin/cron.bash send_mail > /tmp/cron.bash 2>&1

Any error from the command should be in /tmp/cron.bash . What does that tell you?

As a best practice, it is always a good idea to send the STDOUT and STDERR from a cronjob to somewhere. That cronjob might be spewing email to an email address somewhere (Such as 'root'). I am sure that the person who receives root's email doesn't appreciate it.

Personally, I like sending my cronjob output to syslog using /usr/bin/logger. If there is a problem, simply check /var/log/messages.

Community
  • 1
Stefan Lasiewski
  • 23,667
  • 41
  • 132
  • 186
  • Normally, I have it sent to dev/null but for debugging I wanted to see what may show up in syslog. The output of redirecting it to a temporary file is puzzling to me: It's one log line from the python function. When I run the script from the terminal, it continues beyond this line. What is the difference in environment between cron running and me running it from a terminal? – Yuji Tomita Sep 20 '12 at 17:25
  • The cron environment is a very simple version of the user's environment. Dotfiles are not sourced. See the section titled "Things to look out for! Gotchas!" at http://www.pantz.org/software/cron/croninfo.html for some more information. – Stefan Lasiewski Sep 20 '12 at 17:39
  • It's getting as far as running the python function, loading django, etc (and the other commands work too which are the exact same interface - just changes the argument passed to django). so the gotcha's don't seem to be of concern. This is super odd especially since it has worked for years. Why just mail sending, and why only when cron attempts it? I think it might have something to do with the `FileLock` implementation. I am going to see if any updates have been made to it – Yuji Tomita Sep 20 '12 at 17:46