VirtualAlloc - allocating space for big files

Question

I am currently trying to rewrite Binary Diff to support bigger files, as using GetMem for reading files limits the file size (I suppose) and I am not able to read 2 files with a size of 900 MB each.

So I figured, I could use VirtualAlloc, which sadly didn't work out so well yet. The allocation for the first file works fine from the looks of it - once I am trying to allocate memory for the second file, it returns a null pointer. Gotta say I am pretty new to allocating memory, so forgive me if I might have overseen a thread that already answers this question (I've searched the internet for a solution the past 4 hours).

Well, heres the code:

procedure TFileData.LoadFile;
var
  FileHandle: Integer;
  BytesRead: Integer;
  dataPoint : Pointer;
begin
  FileHandle := FileOpen(fName, fmOpenRead or fmShareDenyWrite);
  try
    if FileHandle = -1 then
      Error('Cannot open file %s', [fName]);
    fSize := GetFileSize(FileHandle, nil);
    if fSize = Cardinal(-1) then
      Error('Cannot find size of file %s - may be to large', [fName]);
    if fSize = 0 then
      Error('File %s is empty', [fName]);
     try

        dataPoint := VirtualAlloc(nil,fSize,MEM_COMMIT,PAGE_READWRITE);
        fData := dataPoint;
        BytesRead := FileRead(FileHandle, fData^, fSize);
        if BytesRead = -1 then
          Error('Cannot read from file %s', [fName]);
        if fSize <> Cardinal(BytesRead) then
          Error('Error reading from file %s', [fName]);
     except
      if Assigned(fData) then
        FreeMem(fData, fSize);
      raise;
    end;
  finally
    if FileHandle <> -1 then
      FileClose(FileHandle);
  end;
end;

At the end of the day, I want to use the program to compare two nontext-files binarily of any size and create a Binary Diff from that.

Try [memory-mapped files](https://msdn.microsoft.com/en-us/library/windows/desktop/aa366556%28v=vs.85%29.aspx). — Ondrej Kelle, Mar 02 '15 at 14:32
Memory mapping is not a panacea. You'll encounter the exact same problem if you attempt to map a view of the entire file. Memory mapping can be a useful tool but in order to make progress you need to re-think your algorithm. No amount of trying to implement your current algorithm using different APIs will help. — David Heffernan, Mar 02 '15 at 14:45

David Heffernan · Accepted Answer · 2015-03-02T14:52:36.453

VirtualAlloc won't help. Your call to GetMem for such a large block of memory will be implemented as a call to VirtualAlloc. Of course it doesn't help that your VirtualAlloc code is all wrong. Matching VirtualAlloc with FreeMem is really no good at all. However, since VirtualAlloc isn't the answer, I won't dwell on this.

Your real problem is that your 32 bit process has an address space of between 2GB and 4GB, depending on your OS and whether or not you made your process large-address aware. Finding entire gigabytes of contiguous address space is going to be tricky.

The solution to the problem of not being able to find contiguous blocks of address space is to stop attempting to do so. Don't read the entire file into memory at once. Read the file piece by piece processing smaller blocks.

VirtualAlloc - allocating space for big files

1 Answers1