What's the point of using MPI_Bcast when all ranks can see the data which will be broadcasted?

Question

I'm just thinking about the reason behind using MPI_Bcast because when I do not broadcast integer N to all ranks, they can see the N. Look at the code and its result. Both before and after broadcasting, the integer N is visible to all ranks. so, what's the point here? Also, Does this make sense that using MPI_Bcast changes the order that each rank is called?

#include <iostream>
#include "mpi.h"
using namespace std; 
int main()
{
   MPI_Init(NULL, NULL);
    int rank, size; 
    int N=9 ;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    cout << " Hello from rank : " << rank << " N is: " << N << endl;
    MPI_Bcast(&N, 1, MPI_INT, 0, MPI_COMM_WORLD);
    cout << " Hello from rank : " << rank << " N is: " <<N<<endl;
    MPI_Finalize();

}

Result:

 Hello from rank : 1 N is: 9
 Hello from rank : 3 N is: 9
 Hello from rank : 0 N is: 9
 Hello from rank : 2 N is: 9
 Hello from rank : 0 N is: 9
 Hello from rank : 1 N is: 9
 Hello from rank : 2 N is: 9
 Hello from rank : 3 N is: 9

score 2 · Accepted Answer · answered Mar 06 '21 at 16:51

MPI_Bcast gets useful in cases where N in your example would actually be changed by the broadcasting rank. Consider this instead:

#include <iostream>
#include "mpi.h"

using namespace std;

int main()
{
    MPI_Init(NULL, NULL);
    int rank, size;
    int N=9 ;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    cout << " Hello from rank : " << rank << " N is: " << N << endl;

    // Change N on the rank that will broadcast
    if (rank == 0) {
        N = 18;
    }

    MPI_Bcast(&N, 1, MPI_INT, 0, MPI_COMM_WORLD);
    cout << " Hello from rank : " << rank << " N is: " <<N<<endl;
    MPI_Finalize();
}

Rank 0 changed N and wants to send its own value to all other ranks. It can send it to each rank individually using MPI_Send and MPI_Recv or it can use some form of collective communication, like MPI_Bcast.

MPI_Bcast is the right choice though, because it is optimized for the functionality it provides: sending information from one to all. I advice you check out its communication algorithm to understand.

Since communication is the bottleneck of CPU parallelization, one wants to choose the algorithms for communication carefully. The rule of thumb is that whatever was designed for collective communication, should be used for such, instead of the point-to-point send-receive approach (i.e. rank 0 sends to each other rank individually).

The one-to-all broadcasting is actually a common situation in practice, at least in scientific computing, where for example, one rank needs to read input parameters, prepare the grounds for simulation, and then broadcast some necessary information to all other ranks. Broadcasting may also be needed during the computations, where one rank needs to share its results with a subset of others, or all of them.

Thanks! very useful. Just one more thing, the first argument of MPI_Bcast which is the buffer that N exists there. Does this mean that other processors will look into the address of N in rank 0 and copy the value of N into their own memory? — MA19, Mar 06 '21 at 17:48
Gladly :) I don't think so, MPI uses distributed memory, those ranks may be on different "nodes" - equivalent to completely separated computers, with different memories, even memory layouts. The Bcast will send the value of N to all ranks, and then each will store the value in their own N. Bcast can also be used for arrays, and I assume the principle is the same, it sends values, rather than pointers to sender's memory. — atru, Mar 06 '21 at 17:52
Yeah, you are right. but using & before N has misled me. I was thinking that the address is broadcasting which does not make sense — MA19, Mar 06 '21 at 17:55
I see, all the MPI communication uses pointers as arguments, maybe to make the functionality explicitly clear/the same for both values and arrays. And efficient. You can also do a quick read on the distributed memory paradigm, it becomes clearer when you compare to shared memory parallelization like in OpenMP multithreading. It's actually interesting, then the threads-ranks would literally work on the same bit of memory. — atru, Mar 06 '21 at 18:55

score 1 · Answer 2 · answered Mar 06 '21 at 16:49

In your example the value is hardcoded. There is no point broadcasting a value that's already known.

What if it was computed in, say, rank 0 after startup? Then MPI_Bcast can be used to broadcast it to all other ranks.

    MPI_Init(NULL, NULL);
    int rank, size; 
    int N = 0;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    cout << " Hello from rank : " << rank << " N is: " << N << endl;
    if (rank == 0) {
        N = 9;
    }
    MPI_Bcast(&N, 1, MPI_INT, 0, MPI_COMM_WORLD);
    cout << " Hello from rank : " << rank << " N is: " <<N<<endl;
    MPI_Finalize();

Prints:

 Hello from rank : 0 N is: 0
 Hello from rank : 3 N is: 0
 Hello from rank : 2 N is: 0
 Hello from rank : 1 N is: 0
 Hello from rank : 0 N is: 9
 Hello from rank : 1 N is: 9
 Hello from rank : 2 N is: 9
 Hello from rank : 3 N is: 9

What's the point of using MPI_Bcast when all ranks can see the data which will be broadcasted?

2 Answers2