MPI: How to use MPI_Win_allocate_shared properly

Question

I would like to use a shared memory between processes. I tried MPI_Win_allocate_shared but it gives me a strange error when I execute the program:

Assertion failed in file ./src/mpid/ch3/include/mpid_rma_shm.h at line 592: local_target_rank >= 0 internal ABORT

Here's my source:

    # include <stdlib.h>
    # include <stdio.h>
    # include <time.h>
    
    # include "mpi.h"
    
    int main ( int argc, char *argv[] );
    void pt(int t[], int s);
    
    int main ( int argc, char *argv[] )
    {
        int rank, size, shared_elem = 0, i;
        MPI_Init ( &argc, &argv );
        MPI_Comm_rank ( MPI_COMM_WORLD, &rank );
        MPI_Comm_size ( MPI_COMM_WORLD, &size );
        MPI_Win win;
        int *shared;
        
        if (rank == 0) shared_elem = size;
        MPI_Win_allocate_shared(shared_elem*sizeof(int), sizeof(int), MPI_INFO_NULL, MPI_COMM_WORLD, &shared, &win);
        if(rank==0)
        {
            MPI_Win_lock(MPI_LOCK_EXCLUSIVE, 0, MPI_MODE_NOCHECK, win);
            for(i = 0; i < size; i++)
            {
                shared[i] = -1;
            }
            MPI_Win_unlock(0,win);
        }
        MPI_Barrier(MPI_COMM_WORLD);
        int *local = (int *)malloc( size * sizeof(int) );
        MPI_Win_lock(MPI_LOCK_SHARED, 0, 0, win);
        for(i = 0; i < 10; i++)
        {
            MPI_Get(&(local[i]), 1, MPI_INT, 0, i,1, MPI_INT, win);
        }
        printf("processus %d (avant): ", rank);
        pt(local,size);
        MPI_Win_unlock(0,win);
        
        MPI_Win_lock(MPI_LOCK_EXCLUSIVE, 0, 0, win);
        
        MPI_Put(&rank, 1, MPI_INT, 0, rank, 1, MPI_INT, win);
        
        MPI_Win_unlock(0,win);
        
        MPI_Win_lock(MPI_LOCK_SHARED, 0, 0, win);
        for(i = 0; i < 10; i++)
        {
            MPI_Get(&(local[i]), 1, MPI_INT, 0, i,1, MPI_INT, win);
        }
        printf("processus %d (apres): ", rank);
        pt(local,size);
        MPI_Win_unlock(0,win);
        
        
        MPI_Win_free(&win);
        MPI_Free_mem(shared);
        MPI_Free_mem(local);
        MPI_Finalize ( );
        
        return 0;
    }
    
    void pt(int t[],int s)
    {
        int i = 0;
        while(i < s)
        {
            printf("%d ",t[i]);
            i++;
        }
        printf("\n");
    }

I get the following result:

processus 0 (avant): -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 
processus 0 (apres): 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 
processus 4 (avant): 0 -1 -1 -1 -1 -1 -1 -1 -1 -1 
processus 4 (apres): 0 -1 -1 -1 4 -1 -1 -1 -1 -1 
Assertion failed in file ./src/mpid/ch3/include/mpid_rma_shm.h at line 592: local_target_rank >= 0
internal ABORT - process 5
Assertion failed in file ./src/mpid/ch3/include/mpid_rma_shm.h at line 592: local_target_rank >= 0
internal ABORT - process 6
Assertion failed in file ./src/mpid/ch3/include/mpid_rma_shm.h at line 592: local_target_rank >= 0
internal ABORT - process 9

Can someone please help me figure out what's going wrong & what that error means ? Thanks a lot.

Hristo Iliev · Answer 1 · 2016-05-14T07:44:21.917

5

MPI_Win_allocate_shared is a departure from the very abstract nature of MPI. It exposes the underlying memory organisation and allows the programs to bypass the expensive (and often confusing) MPI RMA operations and utilise the shared memory directly on systems that have such. While MPI typically deals with distributed-memory environments where ranks do not share the physical memory address space, a typical HPC system nowadays consists of many interconnected shared-memory nodes. Thus, it is possible for ranks that execute on the same node to attach to shared memory segments and communicate by sharing data instead of message passing.

MPI provides a communicator split operation that allows one to create subgroups of ranks such that the ranks in each subgroup are able to share memory:

MPI_Comm_split_type(comm, MPI_COMM_TYPE_SHARED, key, info, &newcomm);

On a typical cluster, this essentially groups the ranks by the nodes they execute on. Once the split is done, a shared-memory window allocation can be executed over the ranks in each newcomm. Note that for a multi-node cluster job this will result in several independent newcomm communicators and thus several shared memory windows. Ranks on one node won't (and shouldn't) be able to see the shared memory windows on other nodes.

In that regard, MPI_Win_allocate_shared is a platform-independent wrapper around the OS-specific mechanisms for shared memory allocation.

edited May 14 '16 at 07:44

answered May 14 '16 at 07:39

Hristo Iliev

72,659
12
135
186

Are all processes running on the same node? – Hristo Iliev May 14 '16 at 08:13
@Hristolliev yes in my case they are, but it's just for tests, if I get everything to work, I will be testing on other computers connected by a network... – Reda94 May 14 '16 at 08:16
1

It works with Open MPI except for the fact that you are using `MPI_Free_mem()` to free memory allocated with `malloc()`, which triggers a segfault. Could be a problem in MPICH. Note that you do not need to use `MPI_Get`/`MPI_Put` with shared memory windows and can load from/store into `shared[]` directly. – Hristo Iliev May 14 '16 at 08:36
@Hristolliev but if I load/store directly with shared, I'd be using the local "shared" variable of each process, which is NULL on all processes except process 0 (master). Am I wrong ? – Reda94 May 14 '16 at 08:44
1

Oops, you should use `MPI_Win_shared_query` to obtain a pointer to the part allocated by rank 0. – Hristo Iliev May 14 '16 at 08:55
@Hristolliev Thanks a lot for your help! – Reda94 May 14 '16 at 09:13

alfC · Answer 2 · 2021-02-01T18:56:22.463

There are several problems with this code and the usage. Some of these are mentioned in @Hristolliev's answer.

you have to run all the processes in the same node to have a intranode communicator or use "communicator split shared".
you need to run this code with at least 10 processes.
Third, local should be deallocated with free().
you should get the shared pointer from a query.
you should deallocate shared (I think this is taken care by Win_free)

This is the resulting code:

    # include <stdlib.h>
    # include <stdio.h>
    # include <time.h>
    
    # include "mpi.h"
    
    int main ( int argc, char *argv[] );
    void pt(int t[], int s);
    
    int main ( int argc, char *argv[] )
    {
        int rank, size, shared_elem = 0, i;
        MPI_Init ( &argc, &argv );
        MPI_Comm_rank ( MPI_COMM_WORLD, &rank );
        MPI_Comm_size ( MPI_COMM_WORLD, &size );
        MPI_Win win;
        int *shared;
    
    //    if (rank == 0) shared_elem = size;
    //    MPI_Win_allocate_shared(shared_elem*sizeof(int), sizeof(int), MPI_INFO_NULL, MPI_COMM_WORLD, &shared, &win);
    
       if (rank == 0)
       {
          MPI_Win_allocate_shared(size, sizeof(int), MPI_INFO_NULL,
                                  MPI_COMM_WORLD, &shared, &win);
       }
       else
       {
          int disp_unit;
        MPI_Aint ssize; 
          MPI_Win_allocate_shared(0, sizeof(int), MPI_INFO_NULL,
                                  MPI_COMM_WORLD, &shared, &win);
          MPI_Win_shared_query(win, 0, &ssize, &disp_unit, &shared);
       }
    
        
        if(rank==0)
        {
            MPI_Win_lock(MPI_LOCK_EXCLUSIVE, 0, MPI_MODE_NOCHECK, win);
            for(i = 0; i < size; i++)
            {
                shared[i] = -1;
            }
            MPI_Win_unlock(0,win);
        }
    
        MPI_Barrier(MPI_COMM_WORLD);
    
        int *local = (int *)malloc( size * sizeof(int) );
        MPI_Win_lock(MPI_LOCK_SHARED, 0, 0, win);
        for(i = 0; i < 10; i++)
        {
            MPI_Get(&(local[i]), 1, MPI_INT, 0, i,1, MPI_INT, win);
        }
        printf("processus %d (avant): ", rank);
        pt(local,size);
        MPI_Win_unlock(0,win);
    
        MPI_Win_lock(MPI_LOCK_EXCLUSIVE, 0, 0, win);
    
        MPI_Put(&rank, 1, MPI_INT, 0, rank, 1, MPI_INT, win);
    
        MPI_Win_unlock(0,win);
    
        MPI_Win_lock(MPI_LOCK_SHARED, 0, 0, win);
        for(i = 0; i < 10; i++)
        {
            MPI_Get(&(local[i]), 1, MPI_INT, 0, i,1, MPI_INT, win);
        }
        printf("processus %d (apres): ", rank);
        pt(local,size);
        MPI_Win_unlock(0,win);
    
    
        MPI_Win_free(&win);
     //   MPI_Free_mem(shared);
        free(local);
    //    MPI_Free_mem(local);
        MPI_Finalize ( );
        return 0;
    
    
    }
    
    void pt(int t[],int s)
    {
        int i = 0;
        while(i < s)
        {
            printf("%d ",t[i]);
            i++;
        }
        printf("\n");
    }

MPI: How to use MPI_Win_allocate_shared properly

2 Answers2