increasing the number of CPUs in mpi increses the processing time?

Question

I am new to parallel computing, so I decided to start from hello world compiling with Mpich2. Here is the code:

/* helloworld.c */

#include <stdio.h>

/* You MUST include this for the MPI_* functions */
#include "mpi.h"

int main(int argc, char **argv) {
   int rank;
   char host[150];
   int namelen;

   /* Initialize MPI. This handles mpich-specific command line arguments */
   MPI_Init(&argc, &argv);

   /* Get my rank. My rank number gets stored in the 'rank' variable */
   MPI_Comm_rank(MPI_COMM_WORLD, &rank);

   /* Look up what computer I am running on. Store it in 'host' */
   MPI_Get_processor_name(host,&namelen);

   printf("Hello world (Rank: %d / Host: %s)\n", rank, host);
   fflush(stdout);

   /* Finalize: Close connections to the other children, clean up memory
    * the MPI library has allocated, etc */
   MPI_Finalize();
   return 0;
}

I compile and run it like this:

mpicc helloworld.c -o myhello 
mpirun -nc 2 ./myhello

It works. However I noticed by increasing the number of CPUs the wall clock time increased which I expect it to decrease?! Moreover there is no limitation for the number of CPUs, However my laptop has 5 core but I can set the number of CPUs in the code as much as I want, I expect some error if I exceed the number of real CPUs.

It is not "number of CPUs", it is "number of processes", and you can have way more processes running in the system than you have CPUs (even now you probably have close to a 100 processes running on your system). Can you share your hello world program code? It may help with answering the question, but my guess is that MPI has to create processes and synchronize them, which will increase running time. — bialpio, Nov 11 '14 at 19:25
if you're really new and serious about it, I suggest reading a book. In your case, simply put, n processes doing the same thing will take as much time as 1 process doing that thing. By printing something on each process, you don't distribute any work, and you're actually competing for resources like e.g. I/O. — steabert, Nov 11 '14 at 20:46
It takes more time to launch more processes and to synchronise them at the end. Unless you somehow divide the work among them - which you don't - the total time is only going to increase. That's it. — Hristo Iliev, Nov 11 '14 at 20:50

score 2 · Accepted Answer · answered Nov 11 '14 at 19:29

First off parallel processing in theory should do as you say:

A process done by one should be faster if done by two.

This although is nice to think of in theory, in practice it is actually a completely different story.

Without knowing anything about your project I would think that the program has little parallelizable processing and/or its so fast/small that the message passing that the program has to do is actually slowing it down. Don't forget that its not just the program running there are a lot of other processes done in the background from each core.

What I would advise doing is something that can really show the usefulness of a parrllel process like for example splitting an array into different segments and computing each segment with a different processor (i.e should be a huge array) or reading different text files at the same time and doing some work on them.

As a final note you should really look into Amdahl's law which explains how much a system can speedup with parallel processing.

increasing the number of CPUs in mpi increses the processing time?

1 Answers1