MPI_sendrecv changes loop index with -O1 flag

Question

My code is in a stable version now and I would like to enable at least the O1 optimization level to gain speed. However, I have found a really strange behaviour with a MPI command inside a loop with -O1.

In a preprocessing procedure, each MPI process needs to speak with the others (not necessarily all, that what the SpeakWith array is for) to exchange the number of cells they are dealing with (cpt). That is the goal of the following piece of code :

DO i=1,size(SpeakWith)
    write(*,*) 'BEFORE',i
    CALL MPI_SENDRECV(cpt(1+SpeakWith(i)),1,MPI_INTEGER,SpeakWith(i),ipas,n_recv,1,MPI_INTEGER,SpeakWith(i),ipas,MPI_COMM_WORLD,istat,ierr)
    write(*,*) 'AFTER',i
END DO

I have deliberately added some debugging write commands. When I compile and execute with no optimization flag, everything is fine. With the -O1 flag, I get, for 2 MPI processes :

BEFORE           1
BEFORE           1
AFTER            1
AFTER            0

As you can see, the index loop has been changed during the MPI command. The i variable has been declared with a user-defined kind USER_INT declared in a module :

integer, parameter :: USER_INT = SELECTED_INT_KIND(9)

cpt and SpeakWith have the same kind

I am suspecting a loop optimization that has been activated with the O1 flag, but still I'm stunned that the compiler has modified my code and let such a serious error occurred.

EDIT : A program to debug as requested that runs exactly in the same way than in my code.

PROGRAM test

USE MPI
IMPLICIT NONE

integer(kind=SELECTED_INT_KIND(9)) ::i=0,ierr=0,i_domain=0,nb_domain=1,n_recv=0,istat=0
integer(kind=SELECTED_INT_KIND(9)), dimension(:), allocatable :: SpeakWith
integer(kind=SELECTED_INT_KIND(9)), dimension(:), allocatable :: cpt

CALL MPI_INIT(ierr)
CALL MPI_COMM_RANK(MPI_COMM_WORLD,i_domain,ierr)
CALL MPI_COMM_SIZE(MPI_COMM_WORLD,nb_domain,ierr)

allocate(SpeakWith(1))
allocate(cpt(nb_domain))
cpt(:)=0
IF (i_domain==0) THEN
    SpeakWith(1)=1
    cpt(2)=13226
ELSE
    SpeakWith(1)=0
    cpt(1)=15566
END IF

DO i=1,size(SpeakWith)
    write(*,*) 'BEFORE',i
    CALL MPI_SENDRECV(cpt(1+SpeakWith(i)),1,MPI_INTEGER,SpeakWith(i),1,n_recv,1,MPI_INTEGER,SpeakWith(i),1,MPI_COMM_WORLD,istat,ierr)
    write(*,*) 'AFTER',i
END DO

CALL MPI_FINALIZE(ierr)

END PROGRAM test

are you sure that `SELECTED_INT_KIND(9)` is compatible with `MPI_INTEGER`? also, what is the type of `n_recv`? it might be that the MPI call is receiving more data than `n_recv` can handle and thus overwriting some parts of the memory which might be occupied by other variables... — ewcz, Aug 16 '16 at 15:02
`n_recv` is `SELECTED_INT_KIND(9‌)`. I was wondering that too but I checked with `sizeof` and `SELECTED_INT_KIND(9‌)` is equivalent to a 4-byte integer as well as `MPI_INTEGER` — Coriolis, Aug 16 '16 at 15:18
I am especially concerned about `istat`. We can't make sure it is declared with the right dimension in your code. — Vladimir F Героям слава, Aug 16 '16 at 16:05
I have edited my post. Actually all my integers are declared and initialized at the same time, and indeed if I don't initialize `istat` to zero, it seems to work (I am not sure though) but I don't understand the relevance of initializing `istat` or not in the behaviour of the program. — Coriolis, Aug 16 '16 at 16:21

MPI_sendrecv changes loop index with -O1 flag

0 Answers0