How do you schedule some python scripts to run regularly on a Windows PC?

Question

I have some python scripts that I look to run daily form a Windows PC.

My current workflow is:

1. The desktop PC stays all every day except for a weekly restart over the weekend
2. After the restart I open VS Code and run a little bash script ./start.sh that kicks off the tasks.

The above works reasonably fine, but it is also fairly painful. I need to re-run start.sh if I ever close VS Code (eg. for an update). Also the processes use some local python libraries so I need to stop them if I'm going to update them.

With regards to how to do this properly, 4 tools came to mind:

1. Windows Scheduler
2. Airflow
3. Prefect (https://www.prefect.io/)
4. Rocketry (https://rocketry.readthedocs.io/en/stable/)

However, I can't quite get my head around the fundamental issue that Prefect/Airflow/Rocketry run on my PC then there is nothing that will restart them after the PC reboots. I'm also not sure they will give me the isolation I'd prefer on these tools.

Docker came to mind, I could put each task into a docker image and run them via some form of docker swarm or something like that. But not sure if I'm re-inventing the wheel.

I'm 100% sure I'm not the first person in this situation. Could anyone point me to a guide on how this could be done well?

Note:

• I am not considering running the python scripts in the cloud. They interact with local tools that are only licenced for my PC.

Answer 1

You can definitely use Prefect for that - it's very lightweight and seems to be matching what you're looking for. You install it with pip install prefect, start Orion API server: prefect orion start and once you create a Deployment, and start an agent prefect agent start -q default you can even configure schedule from the UI

For more information about Deployments, check our FAQ section.

Answer 2

It sounds Rocketry could also be suitable. Rocketry can shut down itself using a task. You could do a task that:

1. Runs on the main thread and process (blocking starting new tasks)

2. Waits or terminates all the currently running tasks (use the session)

3. Calls session.shut_down() which sets a flag to the scheduler.

   There is also a app configuration shut_cond which is simply a condition. If this condition is True, the scheduler exits so alternatively you can use this.

Then after the line app.run() you simply have a line that runs shutdown -r (restart) command on shell using a subprocess library, for example. Then you need something that starts Rocketry again when the restart is completed. For this, perhaps this could be an answer: https://superuser.com/a/954957, or use Windows scheduler to have a simple startup task that starts Rocketry.

Especially if you had Linux machines (Raspberry Pis for example), you could integrate Rocketry with FastAPI and make a small cluster in which Rocketry apps communicate with each other, just put script with Rocketry as a startup service. One machine could be a backup that calls another machine's API which runs Linux restart command. Then the backup executes tasks until the primary machine answers to requests again (is up and running).

But as the author of the library, I'm possibly biased toward my own projects. But Rocketry very capable on complex scheduling problems, that's the purpose of the project.

Answer 3

You can use schtasks for windows to schedule the tasks like running bash script or python script and it's pretty reliable too.