c++ read file and store integer in vectors. ends up taking around 5 time more resident memory than actual file size

Question

I need to read in few input files(each contains a 2d matrix of integers) and store them in a vector of 2d vectors. below is code I wrote:

int main(int argc, char *argv[]) {
  /*
    int my_rank;
    int p;
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);
    MPI_Comm_size(MPI_COMM_WORLD, &p);
  */

  std::vector<std::vector<std::vector<int > > > matrices(argc);
  for(int i=1; i<argc; ++i){
      std::string line;
      std::ifstream fp(argv[i]);
      std::vector<std::vector<int> > matrix;

      if (fp.is_open()) {
          while (getline(fp, line)) {
              if(line!=""){
                  //add a new row to file
                  std::vector<int> newRow;
                  //parse each row put the values in the file buffer
                  std::stringstream buff(line);
                  //buffValue is each number in a row
                  int buffValue;
                  while (buff >> buffValue) {
                      newRow.push_back(buffValue);
                  }
                  matrix.push_back(newRow);
              }
          }
      }
      else {
          std::cout << "Failed to read files" << std::endl;
      }
     matrices.push_back(matrix);
  }
  //MPI_Finalize();
  return 0;
}

I have two questions here:

1. when I read in one single file of 175M, the program ended up taking 900M in resident memory. This is a problem because I usually need to read in 4 files with few hundred M's per file. and it will eventually take multiple G's of memory. Is this because of the way I read/store the integers?

2. If I uncomment the lines involve MPI, the resident memory usage goes up to 1.7G, is this normal or I'm doing something wrong here, I'm using MPICH.

Answer 1

Vector-of-vector-of-vector is not an efficient structure to use. You have memory overheads of the vector classes themselves, plus the standard behaviour of push_back.

A vector will grow its memory exponentially when it needs to resize after push_back, in order to meet time complexity requirements. If your vector capacity is currently 10 values, and you add 11 values, then it will most likely resize its capacity to 20 values.

A side-effect of this growth is potential memory fragmentation. Vector memory is defined to be contiguous. The standard allocators do not have a realloc ability, as in C. So, they must allocate more memory elsewhere, move the data, and free the old storage. This can leave holes in memory that your program can't use for anything else. Not to mention reduce cache locality of your data, leading to poor performance.

You would be better off creating a more memory-efficient 2D structure for your matrices, and then push them on to a deque instead of vector. Here's one I prepared earlier ;). At the very least, if you must use vector-of-vector for the matrix, then pre-allocate it using vector::reserve.

If memory is more important to you than I/O, then it's not out of the question to read the file twice. The first time around, you obtain information about matrix sizes and row lengths. Then you pre-allocate all your structures, and read the file again.

Otherwise, using some kind of temporary pool to store your values for a matrix would be acceptable:

std::deque< std::vector< std::vector< int > > > matrices;
std::vector< size_t > columns;  // number of columns, indexed by row
std::vector< int > values;      // all values in matrix

columns.reserve( 1000 );   // Guess a reasonable row count to begin with
values.reserve( 1000000 ); // Guess reasonable value count to begin with

while( getline(fp, line) ) {
    if( line.empty() ) {
        AddMatrix( matrices, columns, values );
    } else {
        std::istringstream iss( line );
        size_t count = 0;
        for( int val; iss >> val; ) {
            values.push_back( val );
            count++;
        }
        columns.push_back( count );
    }
}

// In case last line in file was not empty, add the last matrix.
AddMatrix( matrices, columns, values );

And add the matrix something like this:

void AddMatrix( std::deque< std::vector< std::vector< int > > > & matrices,
                std::vector< size_t > & columns,
                std::vector< int > & values )
{
    if( columns.empty() ) return;

    // Reserve matrix rows
    size_t num_rows = columns.size();
    std::vector< std::vector< int > > matrix;
    matrix.reserve( num_rows );

    // Copy rows into matrix
    auto val_it = values.begin();
    for( size_t num_cols : columns )
    {
        std::vector< int > row;
        row.reserve( num_cols );
        std::copy_n( val_it, num_cols, std::back_inserter( row ) );
        matrix.emplace_back( row );
        val_it += num_cols;
    }

    // Clear the column and value pools for re-use.
    columns.clear();
    values.clear();
}

Finally, I recommend you choose an appropriate integer type from <cstdint> rather than leaving it up the compiler. If you need only 32-bit integers, use int_least32_t. If your data range fits in 16-bit integers, you'll save a lot of memory by using int_least16_t.

Answer 2

I guess you are seeing the combination of 2 effects: Different size of the int + extra memory in vector.

I'm not sure if you are able to see the first effect, though an int takes about 4 bytes of memory (I think they are allowed to make this 8 bytes, though I have not yet seen implementations of it). The character on the other end only takes 1 byte per digit/char + 1 byte for the space. So if you would have a lot of small integer in there, the internal representation will be larger, though if you have a lot of large numbers, it will be smaller. Also check if you are comparing to the size of the file, or the size on disk, as some filesystems support compression!

A next effect that you will most likely notice is the capacity of the vector, as you most likely have many, this can give quite some overhead. In order to not having to realloc every insertion, the std::vector class has a capacity, this is the size it actually uses and will fill in with the objects you add.

Depending on the implementation, the capacity can grow. An example: Doubling the capacity every time you go over it: If you start with a capacity of 10 and you reach a size of 11, the capacity can go to 20, if you reach a size of 21, the capacity can go to 40 ... (Note: This is also the reason that reserving is important, as it will directly give you the right size)

So if you check the capacity and the size of every individual vector, this can be different. If this is really dramatic for you, you can call shrink_to_fit on the vector to realloc the capacity to the actual stored size.

Finally, the size of your program is also influenced by the application itself. I don't think it's gonna be influencing here, though if you happen to link a lot of shared objects and all are loaded during startup, some memory measurements can include the size of these shared objects as part of your programs memory.