0

I am currently participating in a course for efficient programming of supercomputers and multicore processors. Our recent assignment is to measure the latency for the MPI_Send command (thus the time spent sending a zero byte message). Now this alone would not be that hard, but we have to perform our measurements for the following criterias:

  • communication of processes in the same processor,
  • same node but different processors,
  • and for processes on different nodes.

I am wondering: How do i determine this? For proccesses on different nodes i thought about hashing the name returned by MPI_Get_processor_name, which returns the identifier of the node the process is currently running on, and sending it as a tag. I also tried using sched_cpu() to get the core id, but it seems like that this returns a incremental number, even if the cores a hyperthreaded (thus a process would run on the same core). How do i go about this? I just need a concept for determining the localities! Not a complete code for the stated problem. Thank you!

puelo
  • 5,464
  • 2
  • 34
  • 62
  • I think you could use the `KMP_AFFINITY` environment variable (set to `Compact` for same processor, `Scattered` for different processors on the same node) to differentiate the first two. –  May 24 '15 at 13:07
  • @R_Kapp i believe that this is only usable in conjunction with OpenMP. Sadly we are only allowed to use Intel MPI. Thanks anyways – puelo May 24 '15 at 14:57
  • It has the greatest effect with OpenMP (which is why almost all articles written on it are from the perspective of OpenMP), but I believe it should affect how Intel MPI spawns processes as well, if I'm not mistaken. –  May 24 '15 at 14:58
  • 1
    I take that back: Intel MPI uses a different variable, called [`I_MPI_PIN`](https://software.intel.com/sites/products/documentation/hpc/ics/impi/41/lin/Reference_Manual/Environment_Variables_Process_Pinning.htm). Same concept, though. –  May 24 '15 at 15:00
  • I tried it. Is there a way to ensure that this is working? How do i know inside of the program if for example the sender of the message is coming from a process on the same processor? Even with those environment variables. sched_cpu() returns different numbers for all processes. – puelo May 24 '15 at 15:12
  • Unfortunately, [I don't think you can](http://superuser.com/questions/867127/determine-which-cpu-a-process-is-running-on). –  May 24 '15 at 15:14
  • Mhh strange. I will ask my supervisor then. Thanks a lot for your help! – puelo May 24 '15 at 15:15
  • HWLOC can tell you where you are on the node. – Jeff Hammond May 24 '15 at 15:18
  • @Jeff HWLOC_get_cpubind() always returns 0. I give up for now. But thanks for the suggestion! – puelo May 24 '15 at 17:00

1 Answers1

2

In order to have both MPI processes placed on separate cores of the same socket, you should pass the following options to mpiexec:

-genv I_MPI_PIN=1 -genv I_MPI_PIN_DOMAIN=core -genv I_MPI_PIN_ORDER=compact

In order to have both MPI processes on cores from different sockets, you should use:

-genv I_MPI_PIN=1 -genv I_MPI_PIN_DOMAIN=core -genv I_MPI_PIN_ORDER=scatter

In order to have them on two separate machines, you should create a host file that provides only one slot per node or use:

-perhost 1 -genv I_MPI_PIN=1 -genv I_MPI_PIN_DOMAIN=core

You can check the actual pinning/binding on Linux by calling sched_getcpuaffinity() and examining the returned affinity mask. As an alternative, you could parse /proc/self/status and look for Cpus_allowed or Cpus_allowed_list. On Windows, GetProcessAffinityMask() returns the active affinity mask.

You could also ask Intel MPI to report the final pinning by setting I_MPI_DEBUG to 4, but it produces a lot of other output in addition to the pinning information. Look for lines that resemble the following:

[0] MPI startup(): 0       1234     node100  {0}
[0] MPI startup(): 1       1235     node100  {1}
Hristo Iliev
  • 72,659
  • 12
  • 135
  • 186