Python - Every tasks read each names from .txt file (Multiprocessing)

Question

So I have been working abit with multiprocessing and finally got it to work with multiprocessing - Basically now I got it to run so every tasks runs now depending on how many tasks I enter.

def main():


    user_input = 0
    while True:
        try:
            user_input = int(input(Fore.WHITE + 'How many tasks do you wanna run? [NUMBERS] \n' + Fore.RESET))
        except ValueError:
            print(Fore.RED + "Stop being stupid" + Fore.RESET)
            continue
        else:
            HowManyThread = user_input
            print()
            i = 0
            jobs = []
            for i in range(HowManyThread):
                p = multiprocessing.Process(target=info, args=(str(i),))
                jobs.append(p)
                time.sleep(.5)
                p.start()

            for p in jobs:
                p.join()

            sys.exit()

however I was looking through other stackoverflow threads and found

with open(fname) as f:
    content = f.readlines()
# you may also want to remove whitespace characters like `\n` at the end of each line
content = [x.strip() for x in content]

where I have all my names in fname etc:

Barry Alex Sven Mila Jenny etc etc...

However what the multiprocessing is doing though is that it actually reads the same for each tasks and I can't really get a grip what to do now...

Basically what I want to do is that etc...

tasks 1 to read first line tasks 2 to read second line tasks 3 to read third line etc etc...

What could be the best solution for this?

score 1 · Accepted Answer · answered Feb 27 '18 at 14:44

1

You cannot do it. On multiprocessing, every process has own non-shared status.

So every process will ask the operating system to open the file. Every process will start reading the file from the beginning. You can force every process to process only some lines, but every process needs to read all lines, to find where they start and where they end, to count lines.

You example is not something that should be done in multiprocess, but with server (which serve lines) and every multiprocess ask the server for a new line. But this is much more server-client than multiprocess algorithm. Or 'async' or with threads (not sure that read is atomic, so it could make things more complex.)

answered Feb 27 '18 at 14:44

Giacomo Catenazzi

8,519
2
24
32

Oh god! Yeah I was thinking something like that... In that case, What would you recommened to do in that case if I want to run something similar? – Andres123 Feb 27 '18 at 14:47
I don't know (I don't know what you are doing). If bottle neck is disk/network (e.g. processing log file lines and putting every line into a database), async should be a good solution. If every line has much calculation (CPU), server/client is a good solution, and it work nicely to balance load between processes (also at the end). This solution works also on multiple computers. Or just a pre-processor, which split the main files into partial files, to give to the multiprocess.es. It is difficult to tell you. If it is mission critical, you need to test all possibilities. – Giacomo Catenazzi Feb 27 '18 at 15:00

score -1 · Answer 2 · answered Feb 27 '18 at 15:12

-1

You can do this with a process which distribute lines to subprocess. If you are on a UNIX system, you should look for os.fork() and os.pipe().

answered Feb 27 '18 at 15:12

Devilish Spirits

439
4
18

Python - Every tasks read each names from .txt file (Multiprocessing)

2 Answers2