17

A few weeks back I was using std::ifstream to read in some files and it was failing immediately on open because the file was larger than 4GB. At the time I couldnt find a decent answer as to why it was limited to 32 bit files sizes, so I wrote my own using native OS API.

So, my question then: Is there a way to handle files greater than 4GB in size using std::ifstream/std::ostream (IE: standard c++)

EDIT: Using the STL implementation from the VC 9 compiler (Visual Studio 2008). EDIT2: Surely there has to be standard way to support file sizes larger than 4GB.

Joel Coehoorn
  • 399,467
  • 113
  • 570
  • 794
Raindog
  • 1,468
  • 3
  • 14
  • 28
  • STL not especially relevant to this question; remove it? I would suggest iostreams is the name to the part of the C++ standard library you're talking about. – Alastair Nov 16 '08 at 09:02
  • Retagged as Alastair suggested. And to answer, the standard doesn't say anything about how large files should be supported, or how big size_t should be. It's entirely implementation-defined. So which implementation are you using? – jalf Nov 16 '08 at 14:49
  • The question's title needs editing too. – Windows programmer Nov 17 '08 at 00:41

5 Answers5

14

Apparently it depends on how off_t is implemented by the library. 

#include <streambuf>
__int64_t temp=std::numeric_limits<std::streamsize>::max();

gives you what the current max is.

STLport supports larger files.

Eugene Yokota
  • 94,654
  • 45
  • 215
  • 319
5

I ran into this problem several years ago using gcc on Linux. The OS supported large files, and the C library (fopen, etc) supported it, but the C++ standard library didn't. I turned out that I had to recompile the C++ standard library using the correct compiler flags.

KeithB
  • 16,577
  • 3
  • 41
  • 45
  • 1
    I'm with you KeithB, this has _NOTHING_ todo with any problem of MSVC/STL or other, it's a system deployment issue. The source code for the CRT/STL libraries are included with the compiler for a reason, you may need to #define the appropiate preprosessor macro to get your desired functionality. It's NOT that the library implment's one or the other (e.g. *nix using fgetpos or fgetpos64 for >32bit files sized), the library supports BOTH, it's up to the developer to properly use the library. Using STLPort is the same as a recompile of the MSVC STL anyhow, both involve a bit of manual tweaks – RandomNickName42 May 30 '09 at 06:19
2

From the Standard point of view, there is nothing that prevents this. However, in reality, most 32bit implementations use 32bit for std::size_t. Now, the C++ standard mandates that the standard allocator in the C++ Standard Library uses std::size_t as the size quantity. Thus, you are limited to 2^32 bytes of storage for containers, strings and stuff. The situation could be another for std::off_t, i don't know exactly what is going on there.

You have to use the native API of the OS directly, or some library wrapping it, to be able to do that, without having to trust the Standard Library implementations, which are largely implementation-dependent.

Johannes Schaub - litb
  • 496,577
  • 130
  • 894
  • 1,212
1

At least in VS2013, the C++ standard filestream works with large files(>4GBytes) well.

I tested on VS2013 (with update3).

int64_t file_pos = 4LL * 1024 * 1024 * 1024 + 1;
file.seekp( file_pos, SEEK_SET );
assert( file );
cout << "cur pos: " << file.tellp() << endl; // the output is: 4294967297(4GB + 1)

The following link is an additional confirm that it's a bug and has been fixed: https://connect.microsoft.com/VisualStudio/feedback/details/627639/std-fstream-use-32-bit-int-as-pos-type-even-on-x64-platform

for short: Stephan T. Lavavej (Visual C++ Libraries Developer) said

We've fixed it, and the fix will be available in VC11 ... so large file support should work correctly now (regardless of x86/x64 platform)

zhaorufei
  • 2,045
  • 19
  • 18
0

If you can move yourself away from using only standard C++, then you might be interested in boost::iostreams.