3

Important edit : The problem is not what i stated, after manually profiling i understood that when i replace the line : "file >> x >> y >> z;" with the line "file.readline(buffer, size);"

it takes only 0.4 seconds, so the question is entirely different, how to parse the floats from the line, file>>x>>y>>z;

(i don't know if i should delete the question or not, because the original question is not relevant)

=== OLD === After vast research on the internet and stack overflow, i understood that the best way to read large files with c++ is by using memory mapped files.

I have a txt file, 15MB that on each line has 3 float separated by spaces.

I had this code : 

ifstream file(path)
float x,y,z;
while(!file.eof())
  file >> x >> y >> z;

Which could read this file in 9.5 seconds.

In order to read the file faster using stackoverflow users i came up with this code, that if i understand it correctly uses memory mapped files and should read it faster Stream types in C++, how to read from IstringStream?

#include <iostream>
#include <boost/iostreams/stream.hpp>
#include <boost/iostreams/device/mapped_file.hpp>
namespace io = boost::iostreams;

int main()
{
    io::stream<io::mapped_file_source> str("test.txt");
    // you can read from str like from any stream, str >> x >> y >> z
    for(float x,y,z; str >> x >> y >> z; )
        std::cout << "Reading from file: " << x << " " << y << " " << z << '\n';
}

Unfortunately the speed remains the same, still 9.5 seconds.

Any suggestions ? Thanks

Community
  • 1
  • 1
OopsUser
  • 4,642
  • 7
  • 46
  • 71

1 Answers1

2

Streams are slow. Part is because the constraints that apply to them are onerous, part is because implementations have a tendency of being poorly optimized.

Try using Boost.Spirit parsers. While the syntax takes a bit of getting used to and compilation can sometimes be very slow, the runtime performance of Spirit is very high.

Sebastian Redl
  • 69,373
  • 8
  • 123
  • 157