copy part of a file into stream

Question

the global target is using a part of file to get checksum to find duplicated movie and mp3 files, for this goal i have to get a part of file and generate the md5 because whole file size is up to 25 gigs in some cases,if i found duplicates then i will do a complete md5 for avoid any mistake of wrong file deletion i dont have any problem i generating md5 from stream , it will be done with indy components so for first part i have to copy first 1mb of a file

so i maked this function

but the memory stream is empty for all checkes!

function splitFile(FileName: string): TMemoryStream;
 var
    fs: TFileStream;
    ms : TMemoryStream;
 begin
     fs := TFileStream.Create(FileName, fmOpenRead or fmShareDenyWrite) ;
     ms := TMemoryStream.Create;
     fs.Position :=0;
     ms.CopyFrom(fs, 1048576);
     result := ms;
 end;

how can i fix this? or where is my problem?

update1 - (dirty test) :

this code return error stream read error also memo2 show some string but memo3 is empty!!

function splitFile(FileName: string): TMemoryStream;
 var
    fs: TFileStream;
    ms : TMemoryStream;
 begin
     fs := TFileStream.Create(FileName, fmOpenRead or fmShareDenyWrite) ;
     ms := TMemoryStream.Create;
     fs.Position :=0;
      form1.Memo2.Lines.LoadFromStream(fs);
     ms.CopyFrom(fs,1048576);
     ms.Position := 0;
      form1.Memo3.Lines.LoadFromStream(ms);
     result := ms;
 end;

the complete code

function splitFile(FileName: string): TMemoryStream;
 var
    fs: TFileStream;
    ms : TMemoryStream;
    i,BytesToRead : integer;
 begin

     fs := TFileStream.Create(FileName, fmOpenRead or fmShareDenyWrite);
     ms := TMemoryStream.Create;
     fs.Position :=0;
     BytesToRead := Min(fs.Size-fs.Position, 1024*1024);
      ms.CopyFrom(fs, BytesToRead);
     result := ms;
    // fs.Free;
    // ms.Free;
 end;

function streamFile(FileName: string): TFileStream;
 var
    fs: TFileStream;
    ms : TMemoryStream;
 begin
     fs := TFileStream.Create(FileName, fmOpenRead or fmShareDenyWrite) ;
     result := fs;
 end;

 function GetFileMD5(const Stream: TStream): String; overload;
var MD5: TIdHashMessageDigest5;
begin
    MD5 := TIdHashMessageDigest5.Create;
    try
       Result := MD5.HashStreamAsHex(Stream);
    finally
       MD5.Free;
    end;
end;

function getMd5HashString(value: string): string;
var
    hashMessageDigest5 : TIdHashMessageDigest5;
begin
    hashMessageDigest5 := nil;
    try
        hashMessageDigest5 := TIdHashMessageDigest5.Create;
        Result := IdGlobal.IndyLowerCase ( hashMessageDigest5.HashStringAsHex ( value ) );
    finally
        hashMessageDigest5.Free;
    end;
end;

procedure TForm1.Button1Click(Sender: TObject);
var
  Path,hash    : String;
  SR      : TSearchRec;
begin
   if od1.Execute then
  begin
    Path:=ExtractFileDir(od1.FileName); //Get the path of the selected file
    DirList:=TStringList.Create;
    try
          if FindFirst(Path+'\*.*', faArchive  , SR) = 0 then
          begin
            repeat
              if (SR.Size>10240) then
              begin
                hash := GetFileMD5(splitFile(Path+'\'+SR.Name));
              end
              else
              begin
                hash := GetFileMD5(streamFile(Path+'\'+SR.Name));
              end;
                memo1.Lines.Add(hash+' | '+SR.Name +' | '+inttostr(SR.Size));
                application.ProcessMessages;
            until FindNext(SR) <> 0;
            FindClose(SR);
          end;
   finally
     DirList.Free;
   end;
  end;
end;

output:

D41D8CD98F00B204E9800998ECF8427E | eslahat.docx | 13338
D41D8CD98F00B204E9800998ECF8427E | EXT-3000-Data-Sheet.pdf | 682242
D41D8CD98F00B204E9800998ECF8427E | faktor khate ekhtesasi firoozpoor.pdf | 50091
D41D8CD98F00B204E9800998ECF8427E | FileZilla_3.9.0.5_win32-setup.exe | 6057862
D41D8CD98F00B204E9800998ECF8427E | FileZilla_3.9.0.6_win32-setup.exe | 6126536
11210486C9E54E12DA9DF687792257EA | get_stats_of_all_members_of_mu(1).php | 6227
11210486C9E54E12DA9DF687792257EA | get_stats_of_all_members_of_mu.php | 6227
D41D8CD98F00B204E9800998ECF8427E | GOMAUDIOGLOBALSETUP.EXE | 6855616
D41D8CD98F00B204E9800998ECF8427E | harvester-master(1).zip | 54255
D41D8CD98F00B204E9800998ECF8427E | harvester-master.zip | 54180

Answer 1

Here is a procedure that I quickly wrote for you which would alow you to read part of file (chunk) into a memory stream.

The reason why I made this into a procedure and not function is so that it is posible to reuse same memory stream for diferent chunks. This way you avoid all those memory alocations/dealocations and also reduce the chance of introducing the memory leak.

In order to be able to do so you need to pas the memory stream handle to the procedure as variable parameter.

I also adad two more parameters. One for specifying the chunk size (amount of data that you want to read from file) and chunk number.

I also made some rudimentatry safeguards to tell you when you want to read a chunk that is beyond file scope. And also the ability to automatically reduce the size of the last chunk since not all file sizes are multiples of oyur chunk size (in your case not all files are exactly X megabytes in size where X is any valid integer).

procedure readFileChunk(FileName: string; var MS: TMemoryStream; ChunkNo: Integer; ChunkSize: Int64);
var fs: TFileStream;
begin
  fs := TFileStream.Create(FileName, fmOpenRead or fmShareDenyWrite);
  if ChunkSize * (ChunkNo-1) <= fs.Size then
  begin
    fs.Position := ChunkSize * (ChunkNo-1);
    if fs.Position + ChunkSize <= fs.Size then
      ms.CopyFrom(fs, ChunkSize)
    else
      ms.CopyFrom(fs, fs.Size - fs.Position);
  end
  else
    MessageBox(Form2.WindowHandle, 'File does not have so many chunks', 'WARNING!', MB_OK);
  fs.Free;
end;

You use this procedure by calling:

readFileChunk(FileName,MemoryStream,ChunkNumber,ChunkSize);

Make sure you have already created the memory stream before calling this procedure.
Also if you want to reuse the same memory stream multiple times don't forget to set its postion to 0 before calling this procedure othevise new data will be added to the end of the stream and in turn keep increasing the memory stream size.

UPDATE:

After doing some trials I found out that the problem resides in your GetFileMD5 method.

I can't explain exactly why this is happening but if you pass a TMemoryStream to TStream parameter, the TStream parameters simply doesent accept it so the MD5 hashing algorithm the treats it as empty handle.
When I went and changed the parameter type to TMemoryStream instead the code worked but you no longer could pass a TFileStream to GetFileMD5 method anymore so it broke hash generation from entire files that worked before.

SOLUTION:

So after doing some more digging I have a GREAT news for you.

You don't even need to use TMemoryStreams. The "HashStreamAsHex" function can accept two optional parameters which alows you to define the starting point of your data and the size of data block from which you wanna generate the MD5 hash string. And this also works with TFileStream.

So in order to generate MD5 hash string from just small part of your file call this:

MD5.HashStreamAsHex(Stream,StartPosition,DataSize);

StartPositon specifies the inital offset into the stream for the hashing operation. When StartPosition contains a positive non-zero value, the stream position is moved to the specified offset prior to calculating the hash value. When StartPosition contains the value -1, the current position of the stream is used as the initial offset into the specified stream.

DataSize indicates the number of bytes from the stream to include in the hashing operation. When DataSize contains a negative value (<0), the bytes remaining from the current stream position are used for the hashing operation. Otherwise, the number of bytes in DataSize is used. If DataSize is larger than the size of the stream, the smaller of the two values is used for the operation.

In your case for getting the MD5 hash from the first MegaByte you would call:

MD5.HashStreamAsHex(Stream,0,1024*1024);

Now I belive you can modify the rest of your code to get this working as you want to. If not do tell where it stopped and I will help you.

Answer 2

I'm assuming that your code does not raise an exception. If it did you surely would have mentioned that. I also assume that the file is large enough for your attempted read.

Your code does copy. If the call to CopyFrom does not raise an exception then the memory stream contains the first 1024000 bytes of the file.

However, after the call to CopyFrom, the memory stream's pointer is at the end of the stream so if you read from it you will not be able to read anything. Perhaps you need to move the stream pointer to the beginning:

ms.Position := 0;

And then read from the memory stream.

1MB = 1024*1024, FWIW.

---

Update

Probably my assumptions above were incorrect. It seems likely that your code raises an exception because you attempt to read beyond the end of the file.

What you really seem to be wanting to do is to read as much of the first part of the file as possible. That's a two-liner.

BytesToRead := Min(Source.Size-Source.Position, 1024*1024);
Dest.CopyFrom(Source, BytesToRead);