0

I'm running a Python script in a screen session on Digital Ocean. If it crashes, how do I make it automatically restart again?

Vadim Kotov
  • 8,084
  • 8
  • 48
  • 62

1 Answers1

0

I'm not sure screen has this capability to monitor a process for signs of life, but I think something like supervisor will help you. The purpose of screen is only so that you can leave a shell running even though you have disconnected from ssh.

You can download and install (as root) supervisor with pip or easy_install (requires internet access)

pip install supervisor

or

easy_install supervisor

then create and open /etc/supervisord.conf in an editor and populate it with this standard config or create your own by digging into the various config options.

[supervisord]
logfile=/tmp/supervisord.log
logfile_maxbytes=50MB   ; change these depending on how many logs
logfile_backups=10      ; you want to keep
loglevel=info
pidfile=/tmp/supervisord.pid
nodaemon=true
minfds=1024
minprocs=200

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket

[program:myscript]
command=python /path/to/myscript.py ; change to your actual script
autostart=true
autorestart=unexpected ; only restarts when your script has been up for > 1 second and exited with a non-zero exit code.
redirect_stderr=true
stdout_logfile=/var/log/myscript.log

then instead of starting your script directly all you need to do is start supervisor.

/usr/local/bin/supervisord -c /etc/supervisord.conf
esecules
  • 357
  • 1
  • 3
  • 12
  • Thanks! This seems like exactly what I need. Anyways, since my script is on digitalocean, what are the changes I may have to make? I tried leaning about supervisor, but the info seems overwhelming. You've explained it pretty well. –  Dec 08 '15 at 22:09
  • Facing this error: `2015-12-08 17:32:53,762 INFO success: myscript entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2015-12-08 17:32:58,863 INFO exited: myscript (exit status 1; not expected)` –  Dec 08 '15 at 22:33
  • its hard to say without knowing a bit more about your script, is it supposed to be long running? Check the output of `/var/log/myscript.log` (this is where supervisor redirects stdout and stderr) – esecules Dec 08 '15 at 22:35
  • Okay, so the error is because my program saves keys to a file, but it's unable to find the file (which is in the same path as the script). How can that happen? –  Dec 08 '15 at 22:38
  • Correction: the error occurs when trying to open the file, not read from it. Here's the error: `File "/root/folder/myscript/py", line 172, in copy_ids_from_file(stored_ids) File "/root/folder/myscript/py", line 42, in copy_ids_from_file cache_file = open(file_name) IOError: [Errno 2] No such file or directory: 'cachefile'` –  Dec 08 '15 at 22:55
  • Your script is being executed from a different directory. Put `print os.getcwd()` in your script and you'll see what I mean. To fix this always use absolute paths. Or change the cwd to the script's directory. – esecules Dec 08 '15 at 22:56
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/97342/discussion-between-esecules-and-stack). – esecules Dec 08 '15 at 23:03
  • I have another question, if you don't mind answering. When running my script on a screen session, all the statements were printed out in the terminal live as they were executed. Can I make that happen with supervisor? Where can I read them live? I don't want to open log file to read them. Is that possible? –  Dec 08 '15 at 23:18
  • just look at the log file. also you dont need to run supervisor inside of a screen since it is a daemon process and it doesnt need a parent shell to live. any further questions probably deserve a new post. this comment string is getting long. – esecules Dec 08 '15 at 23:21
  • I'm not running it inside screen. Thanks! –  Dec 08 '15 at 23:23