gettimeofday crashes MPI?

Question

I have an mpi program whose run time I'm trying to measure. So I add in a 2 calls to gettimeofday, and that's where everything stops working. For some reason, it only crashes if I have the 2nd call to gettimeofday in there. Here's the message I get:

MPI Application rank 0 killed before MPI_Finalize() with signal 11 srun: error: n32: task0: Exited with exit code 245

Here's the code

   struct timeval starttime;
   struct timeval endtime;

   gettimeofday(&starttime, NULL);

   int numDarts = 1000000000;
   int numWorkers = 2;

   char* args[1];
   if(argc >= 2)
   {
      numWorkers = atoi(argv[1]);
   }
   if(argc >= 3)
        numDarts = atoi(argv[2]);

   MPI_Init(&argc, &argv);
   MPI_Comm_size(MPI_COMM_WORLD, &world_size);

   printf("world size = %i\n", world_size);
   if (world_size != 1)
        printf("Top heavy with management\n");

   int numDartsWorker = numDarts/numWorkers;
   int numDartsMaster = numDarts/numWorkers + (numDarts % numWorkers); //the master computes the leftover
   args[0] = malloc(256 * sizeof(char));
   sprintf(args[0], "%i", numDartsWorker);
  // printf("argument passing to workers: %s\n", args[0]);
   /*
    * Now spawn the workers. Note that there is a run-time determination
    * of what type of worker to spawn, and presumably this calculation must
    * be done at run time and cannot be calculated before starting
    * the program. If everything is known when the application is
    * first started, it is generally better to start them all at once
    * in a single MPI_COMM_WORLD.
    */
  // printf("About to call MPI_Comm_spawn with %i workers...\n", numWorkers);
   int resultLen = 0;

  //the master counts as a worker, hence the -1
   MPI_Comm_spawn("piworker", args, numWorkers-1, MPI_INFO_NULL, 0, MPI_COMM_SELF,
                   &everyone, MPI_ERRCODES_IGNORE);

   double pisum = 0;
   double myresult = dboard(numDartsMaster);
   printf("parent result is %.9f\n", myresult);

   int rc = MPI_Reduce(&myresult, &pisum, 1, MPI_DOUBLE, MPI_SUM, MPI_ROOT, everyone);

   if (rc != MPI_SUCCESS)
        printf("failure on mpi_reduce\n");

   free(args[0]);
   /*
    * Parallel code here. The communicator "everyone" can be used
    * to communicate with the spawned processes, which have ranks 0,..
    * MPI_UNIVERSE_SIZE-1 in the remote group of the intercommunicator
    * "everyone".
    */
   //receive the results
   int i=1;
   MPI_Status status;

   double avgpi = pisum;
   avgpi += myresult; //include master's average in the result.
   avgpi /= numWorkers;

   printf("startTime = %d secs, %d microsecs\n", starttime.tv_sec);


//   gettimeofday(&endtime, NULL);

//   double totalTime = ((double)endtime.tv_sec + (double)endtime.tv_usec/1000000.0f) -
//                    ((double)starttime.tv_usec + (double)starttime.tv_usec/1000000.0f);

 //  printf("Total time: %.8f\n", totalTime);
   printf("With %i workers, %i darts, estimated value of pi is: %.9f\n", numWorkers, numDarts, avgpi);

       MPI_Finalize();


   return 0;
}

I put in a printf call just before the 2nd call to gettimeofday. It only prints anything out if the 2nd call is commented out, otherwise it crashes. I commented out gettimeofday in this example, but that is the call that is causing the crash of mpi. If I uncommented it, it would start crashing again with the error message I mentioned.

I'm wondering if anyone has any insights as to why gettimeofday would be doing this.

It does this because you've trashed memory somewhere, and gettimeofday triggers a crash because of that. Focus on finding buffer overflows and not on gettimeofday — nos, Feb 08 '13 at 23:25
The call to `printf()` just before the commented out call to `gettimeofday()` is missing one argument - there are two `%d` placeholders in the format string but only a single value is begin passed. This could lead to stack corruption. By the way, MPI provides its handy timer function - `MPI_Wtime()`, which returns a (relatively) precise floating point value of the number of seconds elapsed since some fixed point in the past. — Hristo Iliev, Feb 10 '13 at 00:15

score 1 · Answer 1 · answered Feb 08 '13 at 23:02

1

In your code, there is a spawn of thread(s) as in

//the master counts as a worker, hence the -1
MPI_Comm_spawn("piworker", args, numWorkers-1, MPI_INFO_NULL, 0, MPI_COMM_SELF,
               &everyone, MPI_ERRCODES_IGNORE);

and subsequently args is freed as in

free(args[0]);

Is this expected? Won't the free of this memory impact the individual threads?

answered Feb 08 '13 at 23:02

Ganesh

5,880
2
36
54

Nop, it doesn't impact their memory. The program works fine with everything unchanged and with the second gettimeofday() call commented out. (its commmented out in the example I gave, all I have to do is uncomment it to get the program to crash again). Just to be sure, I removed the call to free. It doesn't solve the problem. – NickLokarno Feb 09 '13 at 01:01

score 0 · Answer 2 · answered Feb 09 '13 at 01:39

Thanks to nos, who commented on my question, I was able to find a solution. I got rid of the call to sprintf, which was being used to write onto an argument string for the MPI_Comm_Spawn call, which told the workers how many "darts" to use. Instead I use MPI_Send to send them the information. Seems that somehow sprintf wasn't a problem until I was calling gettimeofday.

gettimeofday crashes MPI?

2 Answers2