Avoid repeated MPI_File_set_view in a datafile with a header

Question

I am modernizing a simple MPI-IO subroutine from https://github.com/LadaF/PoisFFT/blob/master/src/testmpi.f90 (save_vtk() at the very end) somewhat in the spirit of MPI I/O, mix of single- and multiple-process output. My intention is to use this process in a more important program.

I arrived to this code, which produces the correct output (error checking removed):

This is done only one and is insignificant for the performance:

if (master) then
  header =  ... !the header from the link above in a long string
end if
call MPI_Bcast(header_len, storage_size(header_len)/8, &
      MPI_BYTE, 0, glob_comm, ie)

call MPI_Type_create_subarray(3, int(ng), int(nxyz), int(off), &
   MPI_ORDER_FORTRAN, MPI_RP, filetype, ie)

call MPI_type_commit(filetype, ie)

This is done many times:

call MPI_File_open(glob_comm,"out.vtk", &
       IOR(IOR(MPI_MODE_CREATE, MPI_MODE_WRONLY), &
       MPI_MODE_EXCL), MPI_INFO_NULL, fh, ie)

if (ie/=0) then                  !should not happen in serious computations
  call MPI_Barrier(glob_comm, ie)

  if (master) call MPI_File_delete("out.vtk",MPI_INFO_NULL, ie)

  call MPI_Barrier(glob_comm, ie)

  call MPI_File_open(glob_comm,"out.vtk", &
         IOR(MPI_MODE_CREATE, MPI_MODE_WRONLY),&
         MPI_INFO_NULL, fh, ie)
end if

call MPI_File_set_view(fh, 0_MPI_OFFSET_KIND, &
       MPI_CHARACTER, MPI_CHARACTER, "native", &
       MPI_INFO_NULL, ie)

if (master) then
   call MPI_File_write(fh, header, header_len, &
         MPI_CHARACTER, MPI_STATUS_IGNORE, ie)
end if

call MPI_Barrier(glob_comm, ie)
call MPI_File_set_view(fh, header_len, MPI_RP, &
       filetype, "native", MPI_INFO_NULL, ie)

call MPI_File_write_all(fh, buffer, nx*ny*nz, MPI_RP, &
       MPI_STATUS_IGNORE, ie)

call MPI_File_close(fh, ie)

The code first test whether the file already existed and if yes it deletes it and opens again. A full testable code is in a new git branch (https://github.com/LadaF/PoisFFT/blob/new_mpi_io/src/testmpi.f90).

I would like to get rid of the repeated MPI_File_set_view() or at least of the MPI_Barrier() in between, so to set the view for the root process for the header and the chunk of the array straight away and then let the root process to write both the header and the array chunk into the view.

Is that possible?

score 1 · Answer 1 · answered Apr 19 '18 at 16:42

1

I would have the root process just use fortran streamIO with 'replace' to get rid of the MPI_File_delete and multiple view and barrier calls:

if (master) then
  open(file="out.vtk", access='stream', status='replace', newunit=out)
  write(out) header
  close(out)
endif

call MPI_Barrier(...)

call MPI_File_open(...)

call MPI_File_setview(...)

call MPI_File_write_all(...)

call MPI_File_close(...)

There's still one barrier but I don't see a way around it.

answered Apr 19 '18 at 16:42

ptb

2,138
1
11
16

Thanks. I used that open in the previous version. In serious computations the if branch should not be taken too often and is of a minor concern. The real problem is that I have to write the header and then synchronize and set view using the subarray datatype again. A datatype thet would.include the header or some alternative would be great. – Vladimir F Героям слава Apr 19 '18 at 19:34
`MPI_File_set_view()` is a collective operation and `etype` must be the same on all ranks, so I do not see how you can write a header with that (except very convoluted "solutions"). This approach gets rid of `MPI_File_delete()` (note you can use `MPI_File_open()` followed by `MPI_File_set_size()` instead) and to me, looks like the best you can do. – Gilles Gouaillardet Apr 20 '18 at 03:00
It does get rid of the infrequent `MPI_File_delete` but at the price of always opening and closing the file so I don't think it is worth it. I have intentionally did this change, my original code did exactly as above. Normally, the branch should not be taken in serious computations *at all*, it is just for repeated tests so that I don't have to clear the output directory. – Vladimir F Героям слава Apr 20 '18 at 09:31
Is the first File_set_view actually necessary for writing the header when the process just writes at the beginning of a new file? – Vladimir F Героям слава Apr 20 '18 at 10:15
Actually I think you may be able to just use mpi_file_write_at to write the header without creating that first view. – ptb Apr 20 '18 at 12:08
From the MPI 3.1 standard p 492 *The default view is a linear byte stream (displacement is zero, etype and filetype equal to MPI_BYTE).* so if you can safely assume `MPI_BYTE` and `MPI_CHARACTER` are equivalent, then you can skip the first `MPI_File_set_view()` – Gilles Gouaillardet Apr 21 '18 at 02:39

Avoid repeated MPI_File_set_view in a datafile with a header

1 Answers1