5

I am trying to use the mpi_f08 module to do halo exchange on a series of rank 4, 5, and 6 arrays. Previously I used subarray types for this, but ended up with so many that ifort couldn't keep track of all of them and started corrupting them when compiling with -ipo.

I am using code along the lines of 

call MPI_Isend(Array(1:kthird, ksizex_l, 1:ksizey_l, 1:ksizet_l, 1:size5, 1:size6), size, MPI_Double_Complex, ip_xup, 0 + tag_offset, comm, reqs(1))
call MPI_Irecv(Array(1:kthird, 0, 1:ksizey_l, 1:ksizet_l, 1:size5, 1:size6), size, MPI_Double_Complex, ip_xdn, 0 + tag_offset, comm, reqs(2))

(and then later a call to MPI_WaitAll)

ifort 2017 with Intel MPI 2017 gives the following warning for each such line: 

test_mpif08.F90(51): warning #8100: The actual argument is an array section or assumed-shape array, corresponding dummy argument that has either the VOLATILE or ASYNCHRONOUS attribute shall be an assumed-shape array.   [ARRAY]

In spite of this, the halo exchange works fine for rank-4 and -5 arrays. However, when it comes to rank-6 arrays, array data goes to and comes from completely the wrong places, with data from the halo on the sending process (which was not in the array segment passed into MPI_Isend) appearing in the bulk of the receiving process (which was not passed into MPI_Irecv).

Using ifort 2018 and Intel MPI 2019 preview gives an additional error (not warning): 

test_halo_6_aio.F90(60): warning #8100: The actual argument is an array section or assumed-shape array, corresponding dummy argument that has either the VOLATILE or ASYNCHRONOUS attribute shall be an assumed-shape array.   [ARRAY]
call MPI_Isend(Array(1:kthird, ksizex_l, 1:ksizey_l, 1:ksizet_l, 1:size5, 1:size6), size, MPI_Double_Complex, ip_xup, 0 + tag_offset, comm, reqs(1))
-------------------^
test_halo_6_aio.F90(60): error #7505: If an actual argument is an array section with vector subscript and corresponding dummy argument does not have VALUE attribute, it must not have ASYNCHRONOUS / VOLATILE attribute.   [BUF]
call MPI_Isend(Array(1:kthird, ksizex_l, 1:ksizey_l, 1:ksizet_l, 1:size5, 1:size6), size, MPI_Double_Complex, ip_xup, 0 + tag_offset, comm, reqs(1))
^

Three interrelated questions:

  • Is there something incorrect about my syntax in the calls to MPI_Isend and MPI_Irecv that is causing the warnings? How can I fix it so that the warnings are no longer triggered?
  • Is this warning the cause of the array corruption I'm seeing with rank-6 arrays?
  • How can I avoid corrupting rank-6 arrays?

I've put a failing example into this gist.

Ed Bennett
  • 63
  • 8
  • 1
    Which MPI library vendor/version are you using ? Do you observe a similar behavior with `use mpi` bindings ? – Gilles Gouaillardet Apr 20 '18 at 09:20
  • I'm using Intel MPI. I initially tried this approach with the `use mpi` bindings before I switched to defining subarrays, because the data never arrived. My understanding is that until Fortran 2008 creating the subarray made a copy, that was discarded after `MPI_Irecv` wrote into it. – Ed Bennett Apr 20 '18 at 09:28
  • 1
    The understanding is correct unless an MPI datatype for subarrays was used. – Vladimir F Героям слава Apr 20 '18 at 09:47
  • Good to know, thanks. Am I also correct in thinking that this should work with Fortran 2008 and mpi_f08? I've basing this on [the test suite of mpich](https://trac.mpich.org/projects/mpich/browser/test/mpi/f08/subarray/test15.f90?rev=4d8d25184cc87a2694ee808b560f1acb29447c59&order=name) – Ed Bennett Apr 20 '18 at 09:55
  • Strictly speaking, Fortran 2008 *might not* make a copy of the subarray. Intel MPI is MPICH based, and MPICH + Intel Compiler has support for not making a copy, so I guess (because I do not know) that Intel MPI tries not to make a copy of the subarray, and you might have hit a bug. There is no such support in Open MPI (yet) and a copy of the subarray is always made. Bottom line, this is an issue you could/should report to Intel. – Gilles Gouaillardet Apr 20 '18 at 13:40
  • I am surprised that you cannot use MPI datatypes here - you say you need too many? For halo exchange you normally only need one type per dimension which I would have guessed would amount to a few tens of datatypes. Is your application more complicated than you've described here? – David Henty Apr 22 '18 at 12:44
  • The actual application has halo exchange functions for ranks 4, 5, and 6 in double complex, and rank 4 in real, and each defines data types for send and receive, up and down, and directions x, y, and t, giving 48 datatypes (plus a couple extra for MPI-IO). What I was finding was that unit tests worked fine on each halo exchange function individually, but one of the integration tests failed because one group of subarray types seemed to become corrupted. I'll revert the code to that revision and post another question – Ed Bennett Apr 23 '18 at 10:37

0 Answers0