Parallel excution and writing ONCE to a file on python

Question

I have a function that I am running parallelly using joblib. Within that function there is a step that checks for an exception and writes to a file when that exception occurs. I only want to execute this step once (i.e. once on one CPU, not on all CPUs simultaneously) whenever the execption occurs and then continue with parallelly processing the rest of the code.

How can I achieve that? Basically, I want to be able to switch between using all threads and then one thread and then back again to using all threads.

from joblib import Parallel, delayed

def process(x):
    try:
        ...some code
    execpt CustomError as e:
        # Need this portion of the code to be executed on one CPU only
        with open("foo.txt") as f:
            f.write(e)

    ...other code

if __name__ == '__main__':
    Parallel(n_jobs=-1)(delayed(process)(i) for i in range(100))

I just want to make sure I understand your question. Are you saying that you only want to write out the exception error once *for the entire script execution* regardless of how many times the exception occurs? — Booboo, May 16 '23 at 12:40
Exception can occur mutliple times as the loop iterates. Say every 4000 iterations, an exception is bound to occur. — chinex, May 16 '23 at 13:29
Think of it as a token that needs to be refreshed after several iterations. It only needs to be refreshed and written once. — chinex, May 16 '23 at 13:49
I have no idea what you are talking about with your analogy and you still haven't answered my question with a yes or no. If you want to write out the exception more than once during *the entire run*, you need to specify under what conditions it should occur. If the answer is "yes", say so and look at my answer below. — Booboo, May 16 '23 at 13:55

score 1 · Answer 1 · answered May 16 '23 at 12:54

I am assuming that you intend for the exception to be "logged" only once regardless of how many times it occurs. If so, then I would use a sharable (across multiple processes) integer set to 0 to indicate that the exception has not yet been written out. Then when the exception occurs, we inspect the variable to see if it is 0 or not and if it is 0 we write out the exception and ensure that the variable is 1 from that point on:

from joblib import Parallel, delayed
from multiprocessing import Value

def process(exception_written, x):
    try:
        ...  # some code
    except CustomError as e:
        with exception_wrtten.get_lock(): # serialize execution of this block of code
            # Only write out exception if value is 0:
            write_exception = exception_written.value == 0
            # Make sure it is no longer 0:
            exception_written.value = 1
        if write_exception:
            with open("foo.txt") as f:
                f.write(e)

    ... # other code

if __name__ == '__main__':
    exception_written = Value('i', 0) # The exception has not been written
    Parallel(n_jobs=-1)(delayed(process)(exception_written, i) for i in range(100))

Let me know please if this answer works for you. – Booboo May 16 '23 at 14:57 — Booboo, May 16 '23 at 14:57

score 0 · Answer 2 · answered May 16 '23 at 12:15

Maybe you could achieve that using mutexes (look here : Proper use of mutexes in Python or here : https://superfastpython.com/thread-mutex-lock/).

A mutex (from Mutual Exclusion) is a variable shared between parallel processes used to avoid simultaneous access to resources when they should only be accessed once at a time (a DB or a file for example).

In your case, you could listen on all of your processes for a signal that means your exception has been raised, and the quickest process locks the mutex and writes to the file. The others, noticing the mutex is locked, just pass.

Parallel excution and writing ONCE to a file on python

2 Answers2