0

I have a text file which contains a header with the dimensions of a matrix, and then the matrix. Here's an example for a 3x3 matrix:

3 3
1 56 8
12 3 0
9 44 81

I kept getting garbage values with MPI-IO, and found out it only works with binary, not text files.

I thought I'd read in a character stream and convert to integers, but I'm not sure how to approach this problem since the matrix elements have variable number of digits. I'm really not sure how to approach this?

Moody
  • 1,297
  • 2
  • 12
  • 21
  • Why do you want / need to use MPI-IO? Which part of the matrix is each process supposed to read / manage? – Gilles Jan 11 '17 at 08:13
  • @Gilles Each process gets a contiguous block of rows. This is an assignment so I have to use MPI-IO. (I know homework help is frowned upon which is why I am asking for an approach, not a solution!) – Moody Jan 11 '17 at 10:45

2 Answers2

1

Generally you know the type and number of the things written in your binary file (e.g. all integers, 10 int 3 float, etc). You can read as numbers of bytes but MPI binary files are usually read/written as a whole number of types, in your case 9 integers (so number of digits are not important). You open the file and read with something like,

    call MPI_FILE_OPEN(MPI_COMM_WORLD, filename, & 
                   MPI_MODE_RDONLY , MPI_INFO_NULL, fh, ierr)

    bufsize = 3*3
    allocate(buf(bufsize))
    call MPI_FILE_READ_ALL(fh, buf, bufsize, MPI_integer, & 
                           MPI_STATUS_IGNORE, ierr)

For variable matrix sizes, you can use things like MPI_File_get_size to get the size and work out how many elements to read. For mixed data, you can have the first (or last) part of the binary file as a header, which you read first and use to decode the rest of your file. You would still need to know the format of the header and this can be problematic as you break backwards compatibility when you change the code/header format. This is part of the reason for data formats like HDF5 https://support.hdfgroup.org/HDF5/

Ed Smith
  • 12,716
  • 2
  • 43
  • 55
0

text files are tricky because you need to know 'bytes' not 'numbers'. e.g. 1 1 1 is shorter than 10 15 123355.

now, if your convention says "every number will be zero-padded to 6 digits", then you can have each process read from the (size/nprocs)*rank -th

Or, you'll need an indexer to read the file and record at which offset each row of your matrix starts.

Or as you observed, it all becomes much easier with binary data.

Rob Latham
  • 5,085
  • 3
  • 27
  • 44