Is it possible to write data in a output file in parallel

Question

Is it possible to write data on a single output file with more processors. I mean consider some processors have a part of a data (e.g. a matrix) and the whole matrix should be written in a single output file. Is it possible each processors write own parts in parallel (at the same time not one after another)?

Answer 1

Yes it is absolutely possible, and MPI gives you all the tools to do so.

A great introduction to MPI I/O was already linked in the comments. I'm just going to a minimal example to demonstrate it;

#include <stdint.h>
#include <mpi.h>
#include <assert.h>
#include <stdlib.h>
#include <stdio.h>

const int N = 1024ll * 1024 * 256 * 12;
const MPI_Datatype MPI_T = MPI_UINT64_T;
typedef uint64_t T;
const char* filename = "mpi.out";

int main() {
    MPI_Init(NULL, NULL);
    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    assert(N % size == 0);
    T* my_part = calloc(N / size, sizeof(T));
    for (size_t i = 0; i < N / size; i++)
        my_part[i] = i + rank * (N / size);

    MPI_File fh;
    MPI_File_open(MPI_COMM_WORLD, filename,
                  MPI_MODE_WRONLY | MPI_MODE_CREATE, MPI_INFO_NULL,
                  &fh);
    MPI_File_set_view(fh, rank * (N / size) * sizeof(T), MPI_T, MPI_T,
                      "native", MPI_INFO_NULL);

    MPI_Barrier(MPI_COMM_WORLD);
    double begin = MPI_Wtime();
    MPI_File_write_all(fh, my_part, N / size, MPI_T, MPI_STATUS_IGNORE);
    MPI_Barrier(MPI_COMM_WORLD);
    double duration = MPI_Wtime() - begin;
    if (rank == 0)
        printf("Wrote %llu B in %f s, %f GiB/s\n",
               N * sizeof(T), duration,
               N * sizeof(T) / (duration * 1024 * 1024 * 1024));

    MPI_File_close(&fh);
    MPI_Finalize();
}

With very minimal tuning (setting striping size to 12) and 12 ranks, this gets quite reasonable performance on Lustre of ~7.3 GiB/s. Note that this exceeds the raw throughput of a 4xFDR InfiniBand used by the system.

Typically you would use custom data types with such file views, or even more high-level I/O libraries such as HDF5 that work on top of MPI I/O. Getting optimal performance will probably require some site-specific tuning.

In practice, this is sufficient to know, but referring to the discussion on the other answer by user3666197 a few more nitpicky details:

This code snippet expresses concurrent file writes at a quite high-level of abstraction. You get truly parallel I/O by executing this code on an HPC system with a parallel file system. It is absolutely possible that, in a well-tuned configuration, that the bytes written from the different ranks follow entirely different paths and end up on different disks in different storage servers - all in parallel. What matters in your code is to express the concurrency - then you apply tuning to make sure it performs well - which means to allow the storage system to execute it efficiently and in parallel.

Answer 2

Q : Is it possible each processors write own parts in parallel (at the same time not one after another)?

No, it is not. Writing is a process of putting atomic-pieces-of information ( bits to tape, characters to a file-abstraction ) in a pure-[SERIAL] fashion.

If in doubts, take and hold 5 pencils in your hand ( I cannot make it, so it fine to just imagine that one can ) and try to write one word on a paper ( due to "process"-of-writing related circumstances - a singularity of how we write ), there is impossible to "write" 5-independent i.e. different words in this simplified example.

Similarly, in other form of illustration - if you have a typewriter machine ( hope it is not so archaic imagination ) - one can get 5-copies of the same pure-[SERIAL] sequence-of-characters ( thanks to the use of 4-pieces of carbon-copy papers filled between those 5-sheets of office paper ), yet none of these copies will differ from the original - so these are not independent ( like these would be in true-[PARALLEL] processes ) but just a set of replicas, which is productive use of time and resources in producing some paperwork for sending 1-original + 4-copies in some administrative Matrix, but not an example of true-[PARALLEL] writes.

Last but not least, any attempt to use more fingers at once than the one and only the one, for typing on a typewriter ( which puts a pure-[SERIAL] sequence of chars printed on paper ) will produce a mechanical jam, as the process of mechanical type-writing relies on a singular-point, where a character is printed, through a hit of an ink-banner, onto a paper.

Modern filesystems are far from this trivial archetype, yet have a similar concept of producing and maintaining a pure-[SERIAL] representation of a sequence of characters. Even while it is possible to open more filehandles having access "into" this sequence-of-chars, these do not mean one has a chance to make the file-I/O operations de-serialised, the less to take at-once ( as disk-heads are not present at several different locations of the magnetic-disk storage ( the less for tape-device ) at-once and neither the almost-random-access devices, like SSD et al, do not go this wild way, where they lose the control of low-level properties ( wear-leveling, elevator-optimisations, power-limiting and similar low-level device tricks ).