Persistent communication in MPI - odd behaviour

Question

I am solving the coarsest grid of parallel geometric multi grid using jacobi iterations and using Non-blocking calls MPI_Isend() and MPI_Irecv(). There are no problems in this. As soon as I replace the non-blocking communications with persistent communications - the results stop converging at this level and program goes into an infinite loop. The calls MPI_Startall() and MPI_Waitall() always return MPI_SUCCESS. Has anyone faced this problem before ? Please advise.

 Coarsest_grid_solve()
{
MPI_Recv_init(&e_c_old[0][1][1], 1, x_subarray_c, X_DOWN, 10, new_comm, &recv[0]);
MPI_Recv_init(&e_c_old[PXC+1][1][1], 1, x_subarray_c, X_UP, 20, new_comm, &recv[1]);
MPI_Recv_init(&e_c_old[1][PYC+1][1], 1, y_subarray_c, Y_RIGHT, 30, new_comm, &recv[2]);
MPI_Recv_init(&e_c_old[1][0][1], 1, y_subarray_c, Y_LEFT, 40, new_comm, &recv[3]); 
MPI_Recv_init(&e_c_old[1][1][PZC+1], 1, z_subarray_c, Z_AWAY_U, 50, new_comm, &recv[4]);
MPI_Recv_init(&e_c_old[1][1][0], 1, z_subarray_c, Z_TOWARDS_U, 60, new_comm, &recv[5]);

MPI_Send_init(&e_c_old[PXC][1][1], 1, x_subarray_c, X_UP, 10, new_comm, &send[0]);
MPI_Send_init(&e_c_old[1][1][1], 1, x_subarray_c, X_DOWN, 20, new_comm, &send[1]);
MPI_Send_init(&e_c_old[1][1][1], 1, y_subarray_c, Y_LEFT, 30, new_comm, &send[2]);
MPI_Send_init(&e_c_old[1][PYC][1], 1, y_subarray_c, Y_RIGHT, 40, new_comm, &send[3]);
MPI_Send_init(&e_c_old[1][1][1], 1, z_subarray_c, Z_TOWARDS_U, 50, new_comm, &send[4]);
MPI_Send_init(&e_c_old[1][1][PZC], 1, z_subarray_c, Z_AWAY_U, 60, new_comm, &send[5]);  

while(rk_global/r0_global > TOL_CNORM)              
{
coarse_iterations++ ;           

err = MPI_Startall(6,recv);
if(err == MPI_SUCCESS)
    printf("success");

err = MPI_Startall(6,send);
if(err == MPI_SUCCESS)
    printf("success");

err = MPI_Waitall(6, send, MPI_STATUSES_IGNORE);
if(err == MPI_SUCCESS)
    printf("success");

err = MPI_Waitall(6, recv, MPI_STATUSES_IGNORE);
if(err == MPI_SUCCESS)
    printf("success");

//do work here

if(coarse_iterations == 1)
{
    update_neumann_c(e_c_old, PXC, PYC, PZC, X_UP, Y_RIGHT, Z_AWAY_U);
    residual_coarsest(e_c_old, rho_c, PXC, PYC, PZC, X_UP, Y_RIGHT, Z_AWAY_U, hc, rho_temp); 
    r0_local = residual_norm(rho_temp, PXC, PYC, PZC); 
    start_allred = MPI_Wtime();
    MPI_Allreduce(&r0_local, &r0_global, 1, MPI_DOUBLE, MPI_SUM, new_comm); 
    end_allred = MPI_Wtime(); 
    r0_global = r0_global/( (PXC*dims0) * (PYC*dims1) * (PZC*dims2) ); 
    if(rank == 0)
        printf("\nGlobal residual norm is = %f", r0_global);
    rk_global = r0_global;
}           
else
{
    update_neumann_c(e_c_old, PXC, PYC, PZC, X_UP, Y_RIGHT, Z_AWAY_U);
    residual_coarsest(e_c_old, rho_c, PXC, PYC, PZC, X_UP, Y_RIGHT, Z_AWAY_U, hc, rho_temp); 
    rk_local = residual_norm(rho_temp, PXC, PYC, PZC);
    start_allred = MPI_Wtime(); 
    MPI_Allreduce(&rk_local, &rk_global, 1, MPI_DOUBLE, MPI_SUM, new_comm);
    end_allred = MPI_Wtime();  
    rk_global = rk_global/( (PXC*dims0) * (PYC*dims1) * (PZC*dims2) );
    if(rank == 0)
        printf("\nGlobal residual norm is = %f", rk_global);
}

//do dependent work and exchange matrices
}//while loop ends

for(i = 0; i <= 5 ; i++)
{
    MPI_Request_free(&send[i]);
    MPI_Request_free(&recv[i]);
}
}//End coarsest grid solve

Note: Strangely the ghost data becomes zero on alternate iterations. (Just found out - don't know why).

I think it is because I am carrying out Jacobi iterations and exchanging pointers at the end to point to different matrices - it only passes data when the pointer points to the original matrix. — Gaurav Saxena, Nov 20 '15 at 16:12

score 2 · Answer 1 · answered Nov 20 '15 at 16:43

2

When we make a persistent handle for communication, we point to a specific piece of memory that we want to transfer to some other process. Now in Jacobi iterations we need to exchange pointers at the end of the iteration to make the old matrix point to the new updated matrix. Thus, the memory location pointed to by the pointer changes. Thus, the original locations are exchanged. The way around is to define two persistent communication handles. On odd iterations use the first handle and on even iterations use the other handle i.e. alternate them. This solved my problem. This problem also widened my understanding of persistent communications in MPI.

answered Nov 20 '15 at 16:43

Gaurav Saxena

729
1
6
13

Yes, the buffer argument to `MPI_Yyy_init` gets remembered my MPI and then the same buffer is used for all consequently started communications. It's good that you've learned it on your own. An advice - when posting questions in the future, try to first distil your problem into a small fully-functional example code that still reproduces the problem and can be compiled and tested by the members of SO. Usually, you will find the solution on your own while doing so, and if not, you will make the life of those who try to answer your question easier. – Hristo Iliev Nov 20 '15 at 18:36
Many thanks for confirming and the advice. I will follow it from now. – Gaurav Saxena Nov 20 '15 at 19:48

Persistent communication in MPI - odd behaviour

1 Answers1