I have written a python script that runs infinite using loops. The script is started inside a screen session. However sometimes, after a few hours or even days, it breaks down for a reason i dont now, because screen session closes when that happends.
I have also created a "watchdog" script with the following code, which also runs inside a screen session:
from subprocess import check_output
import os
import time
import random
time.sleep(20)
def screen_present(name):
try:
var = check_output(["screen -ls; true"],shell=True)
if "."+name+"\t(" in var:
print name+" is running"
else:
print name+" is not running"
print "RESTARTING"
os.system("screen -dmS player python /var/www/updater.py > /dev/null 2> /dev/null & echo $")
except:
return true
while True:
screen_present("updater")
time.sleep(random.uniform(6, 10))
So when i check my scripts after leaving them running a night or so, i sometimes find, that
- the screen session with the original code is not there anymore, because my script must have thrown an exception but i can't find out which to fix it
- the screen session of my watchdog is marked as "dead"
What would you guys do to find the error and guarantee a stable running?