Which is more efficient way to monitor progress of a long running code? print on screen or log to a text file?

Question

I have a java code that as part of the code it handles a very big matrix (1 million by 4 million) and it takes several hours to run, before it crashes!

I would like to monitor the progress of the code, so know when it runs out of memory and crashes what percentage of matrix was already processes.

I am thinking of adding a command to print on screen a counter (relative to the main operation loop) every 1000 iteration. or logging the counter in a text file.

Is this a good idea? Or will it slow down my already-slow code even further? After all I am adding a condition verification check (to check 1000th iteration) and file-write operation.

Any standard solution in java?

If there is no standard method or function for monitoring the progress, which would be more efficient, writing my own log to a file or printing on screen?

Bonus question: what about in Python? Any internal or standard library for this purspose?

thanks

Answer 1

Without having any context about what this code does, my advice would be that you could place an if condition check for every 1000^th iteration, then does something (print statement, calculation, output to log, check or what have you). This if condition's affect on performance is extremely negligible, unless inside the body of the if condition performs some sort of expensive calculation (modulus expression is not an expensive calculation for computers!).

Python code:

for i in range(100000):  # Loop symbolizing your code.
    if i%1000 == 0:  # Every 1000th iteration do something.
         print "Progress"  # Do something.

That being said unless you're going to be staring at stdout for a few hours in real time; or scrolling though the (what could be very large) stdout log this does not seem like the most efficient idea. I would recommend you write out an error report for this program (function?) while collecting metadata during it's run-time to put in said report, and would propose unit-testing if you aren't already.