0

I am facing some issues while using concurrent.futures Multiprocess module. 

import sys 
import ConfigParser
import os 
import concurrent.futures
from Script_A import get_list, AnalyzeRow # 

rec_list = get_rec_list(MODE)
with concurrent.futures.ProcessPoolExecutor(max_workers=4) as executor:
    for result in executor.map(AnalyzeRow, rec_list):
            print str(result) # This is just for testing

EDIT:

Functions I import from Script_A

def get_rec_list(mode):
    rowBank = []
    current_row = 0

    if mode == 'A':
       setup_logging()
    logging.debug("start")
    create_playlist()
    playlist_url= os.path.join(output_dir+"\\"+test_name, txtplaylist_name)
    if not os.path.exists(playlist_url):
        raise RuntimeError('playlist creation error')
    with open(playlist_url, 'rb') as input_ctrl_csv:
        ref_csv = csv.reader(input_ctrl_csv, delimiter=",")
        for row in ref_csv:
            if current_row > 0:
                row.append(current_row)
                rowBank.append(row)
            current_row += 1

    elif mode == 'B':
        setup_logging()
        logging.debug("start")
        playlist_url = os.path.join(sig_dir, sigplaylist_name)
        if not os.path.exists(playlist_url):
            raise RuntimeError('sig playlist error.')
        with open(playlist_url, 'rb') as sig_list:
            ref_csv = csv.reader(sig_list, delimiter=";")
            for row in ref_csv:
                if current_row > 0:
                    row.append(current_row)
                    rowBank.append(row)
                current_row += 1

    else: 
        raise RuntimeError('invalid run mode')

    return rowBank

def AnalyzeRow(row):
    #This function makes some math to check if the values extracted from the
    csv are inside a certain area range

    generateData(row)
    Error = getError(row)
    return Error

I am aware that Python makes a previous run of the modules that are being imported into my current code, but when I execute the function AnalyzeRow inside the multiprocess function, it recalls to the line where I make the import of AnalyzeRow and get_rec_list functions and runs everything that is in between again, but I don't get any result.

EDIT: When I run my script, it gets to the executor.map(AnalyzeRow, rec_list) line and it jumps back to line from Script_A import get_list, AnalyzeRow so it will execute rec_list = get_rec_list(MODE) 3 more times, since I am using 4 processors. When I execute my code, I get in python interactive window 3 "starts" printed, which corresponds to get_rect_list() function

Is there a better way I could do this execution or is there an error in my code that calls the multiprocess in a wrong way?

user8034399
  • 1
  • 3
  • please fix your indentation. – Mad Physicist Aug 23 '17 at 16:10
  • What does this phrase mean: "Python makes a previous run of the modules that are being imported into my current code"? – Mad Physicist Aug 23 '17 at 16:13
  • What does "it recalls to the line where I make the import of AnalyzeRow and get_rec_list functions and runs everything that is in between again" mean? – Mad Physicist Aug 23 '17 at 16:13
  • Please provide a sample code for `AnalyzeRow` that illustrates the issue. Also, show a sample `rec_list` (doesn't have to be the real one, just something that illustrates the issue/works with the provided function). – Mad Physicist Aug 23 '17 at 16:15
  • Show a sample output that leads you to believe that it is not working. I am having trouble following the narrative portion here, but you can fix that by having more examples. – Mad Physicist Aug 23 '17 at 16:16
  • Until you are able to edit all that into the question, I will have to submit a close vote. – Mad Physicist Aug 23 '17 at 16:16
  • @MadPhysicist I added detailed information to my issue, thanks in advance. – user8034399 Aug 23 '17 at 16:45
  • "so it will execute `rec_list = get_rec_list(MODE)` 3 more times". I am having a lot of trouble believing that statement. – Mad Physicist Aug 23 '17 at 17:51
  • @MadPhysicist Inside get_rect_list(MODE) I have a line that prints "start" plus the current process number. So I get it printed 1 time for the only time I have called this function. Then it is printed 4 times more. – user8034399 Aug 23 '17 at 19:02
  • I am having trouble believing that you pasted `AnalyzeRow` as-is. There is a very obvious syntax error in the code. Does `generateData` or `getError` somehow indirectly call `get_rec_list` again? I will bet you money it does. – Mad Physicist Aug 23 '17 at 21:44
  • This is why you need to post a *minimal* example that reproduces the problem. Do `def get_rec_list(): return [1, 2, 3, 4]`. Then do `def AnalyzeRow(row): return row + 1` or something like that. Does the error still happen in this case? If it does, your multiprocessing is wrong. If it does not, your functions don't do what you think they do. I'm betting on the latter. – Mad Physicist Aug 23 '17 at 21:46
  • My function AnalyzeRow(row) works when I place only one element from element from my rec_list, so when I execute, let's say AnalyzeRow(rec_list[-1]) it has the desired behaviour. – user8034399 Aug 24 '17 at 13:37
  • When you run `AnalyzeRow(rec_list[-1])`, does it print "start"? – Mad Physicist Aug 24 '17 at 13:46
  • the return of rec list has the following structure : [ [ name_of_test, result1, result2, result3], [name_of_test2, result1, result2, result3] ] My function AnalyzeRow(row) works when I place only one element from element from my rec_list, so when I execute, let's say AnalyzeRow(rec_list[-1]), it has the desired behaviour. It also works when I execute the whole script where AnalyzeRow(row) is located. I also use multiprocess in that script that also calls AnalyzeRow(row) too and it works too. I am gonna try making a local function in my current script and see if multiprocess works OK. – user8034399 Aug 24 '17 at 14:07
  • I believe there is something in my multiprocess. I made the following this `rec_list = FunctionA.get_rec_list(MODE)` `rec_list2 = rec_list[-1]` `def test(rec):` ` print rec` `with concurrent.futures.ProcessPoolExecutor(max_workers=4) as executor:` ` for result in executor.map(test, rec_list2):` ` print str(result)` and I still have the the "start" being set into the command window. **EDIT:** It won't let me make code easier to watch in comments. – user8034399 Aug 24 '17 at 14:26
  • First of all, `rec_list[-1]` is not a list but a single item. Second, post your code above in an edit to the question. – Mad Physicist Aug 24 '17 at 18:36

0 Answers0