End of the execution too long using Pool.starmap

Question

I'm executing a paralelized function using Pool.starmap function. The execution of the function it self only takes 6.5 minutes according to tqdm library but the program stays in execution for 20 min more until it finishes. The function is processing and applying filters to some strings in some colums of a pandas dataframe. A different paralelized function could perform better? There is something wrong with starmap function?

Functon to be executed:

def get_best_string_filters(hst, apolnmar, apolnmod, apolnsub, apolnterm, amodnanu, ps, cc, cilindros, combustible, gearbox, year, search_model, search_version, search_container):
    select = table_ecode[(table_ecode.HST == hst)]
    
    year = int(year[-4:])
    
    select = initial_selection(select, ps, cc, cilindros, combustible, gearbox, year)
    
    temp = get_starting_selection(select.copy(), search_model, "HTB")
    if temp.empty:
        search_model, search_version, search_container = find_best_combination(select, search_model, search_version, search_container)
    else:
        select = temp.copy()
        _, search_version, search_container = find_best_combination(select, "", search_version, search_container)
    
    #print(search_model, search_version, search_container)
    
    return [apolnmar, apolnmod, apolnsub, apolnterm, amodnanu, search_model, search_version, search_container]

starmap call:

if not exists("dict_search_ammo_make_version_fixed.npy"):
    params = [(a, b, c, d, e, f, g, h, i, j, k, l, m, n, o) for a, b, c, d, e, f, g, h, i, j, k, l, m, n, o in values_to_change.values]
    with Pool(mp.cpu_count()) as ex:
        array_split_ammo_make_version = ex.starmap(get_best_string_filters, tqdm(params, total=len(params)))
        dict_split_ammo_make_version = array_to_dict(array_split_ammo_make_version)
        # save the dict to disk for faster future executions
        np.save("dict_search_ammo_make_version_fixed.npy", dict_split_ammo_make_version)
else:
    dict_split_ammo_make_version = np.load('dict_search_ammo_make_version_fixed.npy',allow_pickle='TRUE').item()

tqdm outputs 6.5 minutes and a completed status but the script continues to run for 20 long minutes:

See [How to create a Minimal, Reproducible Example](https://stackoverflow.com/help/minimal-reproducible-example). I am not sure how you are doing your timings, but your call to `tqdm` will show how fast your arguments to `starmap` can be generated and not how long it takes for `starmap` to process all the submitted tasks. For that you would be better off using methods `apply_async` with a callback or `imap`. — Booboo, Sep 05 '22 at 13:43
@Booboo I know that tqdm shows the procression when the items are generated but is imposible that the last items take 20 minutes ( i have already timed the time to process the lasts items one by one and isn't even close. The poblem is that starmap stays in execution for a useles amount of time for no reason.... imap function doesn't allow functions with more than one paramater... — kithuto, Sep 05 '22 at 14:03
With `imap` you will be passed a tuple, `tpl` (or whatever name you want to use) as the single argument given that `params` is a list of tuples. But you can unpack this with `a, b, c, ... o = tpl`. Actually, by time you call `starmap` your list `params` had already been generated so I would think your progress bar goes to 100% very quickly. So I am not sure why you say your progress bar takes takes 6.5 minues. — Booboo, Sep 05 '22 at 14:14

Booboo · Accepted Answer · 2022-09-05T14:20:36.147

In the demos below, generator function params simulates generating arguments to worker function foo slowly and foo, which just returns the passed argument, which is either a list when using imap or individual arguments that are the elements of a list.

Using imap

import time

def foo(the_list):
    time.sleep(10)
    return the_list

if __name__ == '__main__':
    from tqdm import tqdm
    from multiprocessing import Pool

    def params():
        for i in range(1, 9):
            time.sleep(1)
            yield list(range(i))

    with Pool() as ex:
        it = ex.imap(foo, params())
        results = list(tqdm(it, total=8))
    print(results)

Using apply_async

import time

def foo(*args):
    time.sleep(10)
    return args

if __name__ == '__main__':
    from tqdm import tqdm
    from multiprocessing import Pool

    def params():
        for i in range(1, 9):
            time.sleep(1)
            yield list(range(i))

    def my_callback(result):
        bar.update(1)

    with Pool() as ex, tqdm(total=8) as bar:
        results = []
        async_results = [ex.apply_async(foo, param, callback=my_callback) for param in params()]
        results = [async_result.get() for async_result in async_results]
    print(results)

imap with fixed sized tuples

import time

def foo(tpl):
    time.sleep(10)
    # unpack:
    a, b, c, d, e, f, g, h = tpl
    return (a + b) * (c +  d) * (e + f) * (g + h)


if __name__ == '__main__':
    from tqdm import tqdm
    from multiprocessing import Pool

    def params():
        for i in range(1, 9):
            time.sleep(1)
            yield list(range(8))

    with Pool() as ex:
        it = ex.imap(foo, params())
        results = list(tqdm(it, total=8))
    print(results)

imap doesn't allow to execute functions with multiple parameters. I will try to time apply_async to see how it performs. — kithuto, Sep 05 '22 at 14:10
I have updated answer to show how you would use `imap` with fixed-size tuples being passed to the worker function. The advantage of using `imap` or `apply_async` is that if your values being passed to your worker function are being generated slowly, they get submitted to the pool for execution as each new value is generated. With `starmap` all the values must be generated first before they are submitted to the pool for execution so there is no overlap between generating values and processing them. — Booboo, Sep 05 '22 at 14:21
Ahh okey just using tuples it can use multiple paramaters i see. Lets see how it performs thanks! — kithuto, Sep 06 '22 at 07:39

End of the execution too long using Pool.starmap

1 Answers1