0

I am new to python and Parallel Python. The problem is I have 4 jobs to be done: generation of 4 masks, multiplication of them with my input image, followed by further processing. Following is the piece of the code written for parallel processing.

inputs = range(4)
jobs = [(inpt, job_server.submit(PP, (inpt,input_data,size,(imageMultiply,blockCounter,imageQuantizer ), ("numpy","Image"))) for inpt in inputs]
job_server.print_stats()
for inpt, job in jobs:
  print "No of blocks in ", inpt, "is", job() ## accessing the result of pp

The output that I get is :

Starting pp with 4 workers
Job execution statistics:
 job count | % of all jobs | job time sum | time per job | job server
         4 |        100.00 |       0.0000 |     0.000000 | local
Time elapsed since server creation 0.0219678878784
4 active tasks, 4 cores

No of blocks in  0 is 52
No of blocks in  1 is 61
No of blocks in  2 is 104
No of blocks in  3 is 48

I cant understand that if its not processing simultaneously, still I am able to get the desired output, but the time taken is too big which is why I want to use pp. Please help me with this so that I can successfully reduce the time. Thanks in advance...

divibisan
  • 11,659
  • 11
  • 40
  • 58
pypro
  • 447
  • 1
  • 7
  • 13
  • You should also call `print_stats()` after all the jobs are joined with `job()`. Then you can compare elapsed times measured with and without Parallel Python. When you say "the time taken is too big," no one can tell how big it is actually; 1 min, 1 hour, 1 day, ... – nodakai Jan 23 '14 at 16:18
  • @nodakai I wrote print_stats() after the last line and got following results: Job execution statistics: job count | % of all jobs | job time sum | time per job | job server 4 | 100.00 | 16.0401 | 4.010028 | local Time elapsed since server creation 4.04183793068 0 active tasks, 4 cores – pypro Jan 23 '14 at 16:39

1 Answers1

0

From your print_stats() output (reformatted here,)

Job execution statistics:
 job count | % of all jobs | job time sum | time per job | job server
         4 |        100.00 |      16.0401 |     4.010028 | local
Time elapsed since server creation 4.04183793068
0 active tasks, 4 cores

everything seems to be fine. You have 4 CPU cores; you could complete 4 jobs within 4 seconds after the creation of the job server; the system consumed 16 CPU seconds in total to get all the jobs done.

You may want to try top -H, htop or Windows Sysinternals Process Monitor to observe CPU consumption in real time.

nodakai
  • 7,773
  • 3
  • 30
  • 60
  • thanks for bringing that to my notice..But I have few doubts: 1) As you have pointed out, " the system consumed 16 CPU seconds in total to get all the jobs done", doesn't that mean that without pp the CPU would have taken 16 sec and after using 4 workers the system takes only 4 sec in total to get all the jobs done?? 2) In each of the 4 jobs that I have assigned, I display a processed image of my input image.So after the execution of my code, all the 4 images should be shown all at once due to use of pp. But I see them pop sequentially.What do I make out of this?plz help me understand – pypro Jan 24 '14 at 16:36
  • 1) Yes, that's what I meant. Can you try passing `ncpus=1` when you instantiate a `Server` object to see how long it takes with 1 CPU? 2) No idea, maybe OS has to serialize disk I/O ops when you access the output image files? Again, why don't you just run CPU usage monitors like shown above? What platform do you use, Linux, OS X or Windows? – nodakai Jan 24 '14 at 18:10