How can I pause a long running loop?

Question

I have a question about pausing / resuming a long running program. I'll use Python in this example, but it could be for any programming language really.

Let's say I want to sum up all the numbers to a billion for example

results = []

for i in range(1000000000):
    results.append(i + i*2)

with open('./sum.txt', 'w') as outfile:
    for r in results:
        output.write(r+'\n')
outfile.close()

That's a very simple program and in the end I would like to write the values out to a file.

Let's say I start this script and it's taking hours and I want to shut down my computer. Now I could wait but what are some ways that I could stop this process while keeping the data in tact?

I've though about opening the file before the loop and inside the loop, instead of appending to the results list, I could just write [append] to the opened file but that seems like still if I quit the program, the file never closes and the data could get corrupted. Even if I just write to the file in the loop then directly close the file after the write operation, couldn't that still cause data corruption or something?

What are some ways that I could tackle a problem like this. Where I'd like to be able to stop a program and later, restart the program and it could continue from where it left off?

If you have a long running script that does most of the work on the inner part of a loop.

@MisterMiyagi Can you please specify what your actual question is? Creating safe-points inside the loop is the way to go if you want to preserve the loop's state. Whether file I/O is safe against corruption doesn't depend on your application so much as operating/file system. In Linux, it's safe to write to a temporary file, then rename it; files are also closed automatically when a program exits.

I was more interested in a safe place inside the loop. What are some recommended ways to do that in a program that's structured like the one above.

Use your hardware/OS to do it, hibernate, or put the whole thing in a virtual machine and save the machine state instead of turning it off. — Maximilian Peters, Oct 05 '16 at 20:28
I was thinking something along the lines of putting the loop inside of a `try` block and then `except KeyboardInterrupt: outfile.close()`so that data would be intact if you force closed the program during the loop. The 'picking up where you left off' idea is mind-boggling for me, though! — Mangohero1, Oct 05 '16 at 20:30
You could change the code to write both the total sum as well as the last number that was added as a comma separated value. Then change the code so that it checks the last number added and continues from there, or starts from 0 if that field is empty. — RBuntu, Oct 05 '16 at 20:32
Can you please specify what your actual question is? Creating safe-points inside the loop is the way to go if you want to preserve the loop's state. Whether file I/O is safe against corruption doesn't depend on your application so much as operating/file system. In Linux, it's safe to write to a temporary file, then rename it; files are also closed automatically when a program exits. — MisterMiyagi, Oct 05 '16 at 20:37
You need to save and restore the state of the program somehow. There are often OS related ways of saving the state of the entire computer (i.e. entering a "sleeping" or "hibernating" state), although that may be overkill to preserve one process. A more lightweight approach would be to add functions or methods to you program that can used to tell it to do these things itself (using by writing and later reading back a file contain information about just its state). — martineau, Oct 05 '16 at 20:58
@MisterMiyagi I am interested in creating a safe point inside the loop so that if I terminate the application, I could restart the program and continue from where I left off. — user1610950, Oct 06 '16 at 04:11
A solution *inside the loop* will only work if the loop will run on, but will not perform the addition, since any event that will actually *stop* or pauze the loop will also stop checking the condition for it to continue. There are thinkable solutions, but possible solutions will be language specific in any way, so *"it could be for any programming language really."* will lead to "too broad". — Jacob Vlijm, Oct 06 '16 at 05:59
@JacobVlijm that's why I tagged the question python and also said using python. — user1610950, Oct 06 '16 at 06:22
Edited the loop, it had a minor bug, skipping one loop on resume. — Jacob Vlijm, Oct 06 '16 at 09:41

score 1 · Answer 1 · answered Oct 05 '16 at 20:38

1

Ctrl+C won't corrupt the file. I'm not sure how sophisticated you need to be, but a simple solution would be to be able to give the program an input of where to resume from:

def main(start=0):
    with open('./sum.txt', 'w') as outfile:
        for i in range(start, 10000000):
            outfile.write(str(i) +'\n')


if __name__ == "__main__":
    import sys
    try:
        start = int(sys.argv[1])
    except IndexError:
        start = 0
    main()

Then run this like to resume from 1000:

python foo.py 1000

answered Oct 05 '16 at 20:38

Ceasar

22,185
15
64
83

I up voted this answer but selected the other one because I think both of these answers will help solve the problem. I can check if some file exists and if it doesn't create it. To resume the program I just rerun the program, if the file exists, get the last value, init the loop and continue. Thanks for the additional information. – user1610950 Oct 06 '16 at 07:02

score 1 · Accepted Answer · edited Jun 20 '20 at 09:12

Create a condition for the loop to either continue or hold

A possible solution would be in the script, periodically checking for a "trigger" file to exist. In the example below, the loop checks once in hundred cycles, but you could make it 500 for example.

If and as long as the file exists, the loop will hold, checking again every two seconds.
While this works fine, you will have to accept the fact that it slightly slows down the loop, depending on n; larger n will decrease the effect. ANY solution inside the loop will however have at least some effect on the loop.

Although this is on python, the concept should be possible in any language, and on any os.

The print functions are only to show that it works of course:)

The edited loop:

import os
import time

results = []
n = 100

for i in range(1000000000):
    results.append(i + i*2)
    print(i)
    if i%n == 0:
        while os.path.exists("test123"):
            time.sleep(2)
            print("wait")   
        else:
            print("go on")
        
with open('./sum.txt', 'w') as outfile:
    for r in results:
        output.write(r+'\n')
outfile.close()

The result, if I create the trigger file and remove it again:

4096
4097
4098
4099
wait
wait
wait
wait
wait
wait
go on
4100
4101
4102

This along with the answer provided below can be combined to give a pretty robust solution that should be pretty language independent. Thank you! — user1610950, Oct 06 '16 at 07:03

How can I pause a long running loop?

2 Answers2

Create a condition for the loop to either continue or hold

The edited loop: