Interleaving binary data from different processors on MPI-IO

Question

I'm trying to write a binary file using MPI I/O (MPI-2.0, mpich2).

Below is a minimal example where 2 files 'chars' and 'ints' should be printed as '0123...' and 'abcd...' respectively.

#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) 
{
    // Initialize the MPI environment
    MPI_Init(NULL, NULL);

    // Get the number of processes
    int world_size;
    MPI_Comm_size(MPI_COMM_WORLD, &world_size);

    // Get the rank of the process
    int world_rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

    // MPI I/O
    MPI_File file;
    MPI_Status status;
    int offset = world_rank;

    // MPI I/O with integers
    MPI_File_open (
        MPI_COMM_WORLD,     // MPI_Comm comm
        "ints",             // char *filename
    MPI_MODE_CREATE |       // int amode
    MPI_MODE_RDWR,
    MPI_INFO_NULL,          // MPI_Info info
    &file                   // MPI_File *fh
    );
    int buf1 = world_rank;
    MPI_File_write_at (
        file,               // MPI_File fh
        offset,             // MPI_Offset offset
        &buf1,              // void *buf
        1,                  // int count
        MPI_INT,            // MPI_Datatype datatype
        &status             // MPI_Status *status
    );
    MPI_File_close (&file);

    // MPI I/O with chars
    MPI_File_open (
        MPI_COMM_WORLD,     // MPI_Comm comm
        "chars",            // char *filename
    MPI_MODE_CREATE |       // int amode
    MPI_MODE_RDWR,
    MPI_INFO_NULL,          // MPI_Info info
    &file                   // MPI_File *fh
    );
    char buf2 = 'a' + (char)world_rank;
    MPI_File_write_at (
        file,               // MPI_File fh
        offset,             // MPI_Offset offset
        &buf2,              // void *buf
        1,                  // int count
        MPI_CHAR,           // MPI_Datatype datatype
        &status             // MPI_Status *status
    );
    MPI_File_close (&file);


    // Finalize the MPI environment.
    MPI_Finalize();
    return 0;
}

I get the right result with chars,

> od -c chars
0000000   a   b   c   d
0000004

but with integers it works only if np = 0. For np > 0, I get results which I don't understand:

np = 2:

> od -i ints
0000000         256           0
0000005

np = 3:

> od -i ints
0000000      131328           0
0000006

etc.

Is my code wrong, or is 'od -i' simply inappropriate for displaying a binary file with integers?

Thanks,

Juri

I need to develop a code for my PhD which produces lots of data on a computer cluster. I don't have any experience with binary files yet, but ASCII is not supported by MPI I/O — Juri Romazanov, Mar 19 '15 at 14:09
Hm, it's designed for serial, not parallel applications. If you have really large data blocks to write, I think parallel I/O is the only feasible option... But I could try C fread/fwrite as a start to learn about binary data files, so thanks for the hint — Juri Romazanov, Mar 19 '15 at 14:14
Got no idea about that, although it seems a little counter intuitive that you could write to Hard Drive in parallel mode. There are mechanical widgets involved in the process. But hey, I'm only speculating. — The Vivandiere, Mar 19 '15 at 14:15
Besides, if you just want a working solution without sweating it, you should consider copying the data from GPU to CPU (RAM) and write the files using C API — The Vivandiere, Mar 19 '15 at 14:17
@user3670482 - binary files are usually what you want for large MPI computations, and the systems one generally uses MPI on have parallel file systems. This is a perfectly sensible [MCVE](http://stackoverflow.com/help/mcve) for a MPI-IO question. — Jonathan Dursi, Mar 19 '15 at 14:20
@JonathanDursi Had no idea about parallel file systems. Very interesting. Thanks! — The Vivandiere, Mar 19 '15 at 14:30

Jonathan Dursi · Accepted Answer · 2015-03-19T14:25:29.297

Unless you've set a file view with an elementary type, the offset index is actually in units of bytes - that is, the default etype is single bytes. This is actually very sensible, but it would be helpful if it were documented in the manpages.

So changing this line:

MPI_File_write_at (
    file,               // MPI_File fh
    offset,             // MPI_Offset offset

to this:

MPI_File_write_at (
    file,               // MPI_File fh
    offset*sizeof(int), // MPI_Offset offset

Gives you the answer that you want:

$ od -i ints 
0000000           0           1           2
0000014

On the other hand, if you'll only be using integers in this file, easier is to set a file view:

MPI_File_set_view( file, 0, MPI_INT, MPI_INT, "native", MPI_INFO_NULL);
MPI_File_write_at (
    file,               // MPI_File fh
    offset,             // MPI_Offset offset

Also works. And in general, you'll want to use different file views rather than explicit offsets for each processor.

Interleaving binary data from different processors on MPI-IO

1 Answers1