ifstream, getline, and istringstream slow performance on big file in c++

Question

I want to ask about my code below: (The code below basically read an input file named inputVelocity.dat with format stated in the code. The code reads with istringstream to pass each value to each particular arrays)

std::ifstream inputVelocity("input/inputVelocity.dat");
std::string lineInputVelocity;
while (std::getline(inputVelocity, lineInputVelocity))  {
  std::istringstream issVelocity(lineInputVelocity);
  double a, b, c, d, e;

  if (!(issVelocity >> a >> b >> c >> d >> e))  {
    std::cout << "ISS ERROR" << std::endl;
  }

  for (int k=0; k<=nz+1; k++)  {
    for (int j=0; j<=ny+1; j++)  {
      for (int i=0; i<=nx+1; i++)  {
        ux[i][j][k]     =   a;
        uy[i][j][k]     =   b;
        uz[i][j][k]     =   c;
        pressure[i][j][k]   =   d;
        temperature[i][j][k]    =   e;
      }
    }
  }
}

inputVelocity.close();

The code is fine when reading around 20000 lines, but when I change the file into around 1.6 million lines, the code run very slow even on a server.

I did std::cout on each getline loop and it read like 5 lines/second, with around 1.6 million lines.

I have found some related questions here but still can't understand what's the problem source and how to fix it. Anyone can help? Thank you.

first of all, are you sure that you want to invoke three 'for' loops for each line? do you know that you have complexity O(LINE COUNT x NX x NY x NZ) for your load procedure? — AnatolyS, May 17 '16 at 10:21
@AnatolyS so you mean that was the case? How can I improve the code? because all of my variable is 3-dimensional vector. — mfakhrusy, May 17 '16 at 10:29
I cannot understand what you are doing inside the while loop. Try to explain yourself what you want to do. — AnatolyS, May 17 '16 at 10:35
Is this code correct at all? You appear to be reading and parsing lines from a file, and then for each one setting EVERY element of the arrays to that value, overwriting the previous value each time. — jcoder, May 17 '16 at 10:36
I feel so dumb, yes indeed, I just realized that. Thanks all for giving me the hints :) It's like I need to take a coffee first. I will answer my own question after this. for @AnatolyS I want to read a file and insert it to variable which stated above, that's it. Thank you for your first comment! — mfakhrusy, May 17 '16 at 11:09

score 0 · Accepted Answer · answered May 17 '16 at 11:45

I changed the code into this:

std::ifstream inputVelocity("input/inputVelocity.dat");
std::string lineInputVelocity;
int getI, getJ, getK;
getI = 0;
getJ = 0;
getK = 0;
while (std::getline(inputVelocity, lineInputVelocity))  {
  std::istringstream issVelocity(lineInputVelocity);
  double a, b, c, d, e;

  if (!(issVelocity >> a >> b >> c >> d >> e))  {
    std::cout << "ISS ERROR" << std::endl;
  }
  std::cout << getI << " " << getJ << " " << getK << std::endl;
  ux[getI][getJ][getK]              =   a;
  uy[getI][getJ][getK]              =   b;
  uz[getI][getJ][getK]              =   c;
  pressure[getI][getJ][getK]        =   d;
  temperature[getI][getJ][getK]     =   e;

  getK = getK + 1;

  if (getK == nz+2)  {
    getJ = getJ + 1;
    getK = 0;
  }

  if (getJ == ny+2)  {
    getI = getI + 1;
    getJ = 0;
  }

}

inputVelocity.close();

And that worked really well :) If anyone has a more efficient solution, I would be very glad to see! :)

ifstream, getline, and istringstream slow performance on big file in c++

1 Answers1