TidHTTPServer "Out of memory" on large file upload

Question

I'm using Delphi 10.3.1 and Indy TIdHTTP / TIdHTTPServer

I created a client / server application to archive files. The client uses a TIdHTTP component, the code is something like this:

procedure TForm1.SendFileClick (Sender: TObject);
var
    Stream: TIdMultipartFormDataStream;
begin
    Stream: = TIdMultipartFormDataStream.Create;
    try
       Stream.AddFormField ('field1', 'hello world');
       Stream.AddFile ('field2', 'c:\temp\gigafile.mp4');
       idHTTP.Post ('http://192.168.1.100:1717', Stream);
    finally
       Stream.Free;
    end;
end;

The server uses a TIdHTTPServer component. Everything seemed to work perfectly until I uploaded very large video files (>= 1GB), because I got the error "Out of memory".

By debugging, I saw that I get the error in function PreparePostStream (line 1229 of the IdCustomHTTPServer unit) when it calls LIOHandler.ReadStream, the event OnCommandGet is not fired yet.

The function LIOHandler.ReadStream goes wrong when it runs AdjustStreamSize (line 2013 of the IdIOHandler unit)

In my last test, with a large video file, in the AdjustStreamSize function, the value of ASize was 1091918544 and I got the error during the execution of

AStream.Size line: = ASize

I think that the origin point of the error is in the System.Classes unit in the following procedure when at the SetPointer ... line.

procedure TMemoryStream.SetCapacity (NewCapacity: NativeInt);
{$ IF SizeOf (LongInt) = SizeOf (NativeInt)}
begin
  SetPointer (Realloc (LongInt (NewCapacity)), FSize);
  FCapacity: = NewCapacity;
end;

I read many articles on the web but I didn't understand if there is something wrong in my code. How can I solve it or is there a limit to the size of the files I can upload with TIdHTTPServer?

@J: TIdMultipartFormDataStream is client side, his problem is that the http server uses a Memorystream to get the file... — whosrdaddy, Jun 06 '19 at 13:42
The fastest solution to your problem is switch to 64bit application (and offcourse 64bit server- with enough RAM). The longterm solution will have to be found on the Indy side. Maybe @RemyLebeau can comment on this... — whosrdaddy, Jun 06 '19 at 13:44
Possible workaround: use a plain binary HTTP payload instead of multipart, and direct the HTTP input stream of the TIdHTTPRequest info to a TFileStream — mjn, Jun 06 '19 at 14:24
@mjn `TIdFormDataField` defaults to `quoted-printable` only for text fields. It defaults to `binary` for files and streams — Remy Lebeau, Jun 06 '19 at 20:25

score 2 · Answer 1 · answered Jun 06 '19 at 20:22

2

By default, TIdHTTPServer receives posted data using a TMemoryStream, which will obviously not work well for such large files. You can use the server's OnCreatePostStream event to provide an alternative TStream object to receive into, such as a TFileStream.

answered Jun 06 '19 at 20:22

Remy Lebeau

555,201
31
458
770

I did it `VPostStream := TFileStream.create(myUniqueTempFileName), fmCreate);` , it work fine with "small" files, but with large ones (50MB) I get a Socket Error 10053. I'm testing my client and server app on the same PC and nothing changes with Antivirus and firewall disabled. The error raises in unit IdCustomHTTPServer at line 2153 `FConnection.IOHandler.Write(...);` – Davide Jun 08 '19 at 08:38
@Davide that is not helpful info. 10053 means the connection was aborted unexpectedly, which can happen for any number of reasons. Also, in the latest `IdCustomHTTPServer.pas`, line 2153 is not a call to `IOHandler.Write()`. What version are you using? – Remy Lebeau Jun 08 '19 at 14:55
my IdCustomHTTPServer.pas is rev 1.42 dated 3/14/05 (the one installed with Delphi 10.3.1 Professional), but in this file your most recent comment is on 11/23/2014: Per RFC 2616 Section 4.3 (line 2133) – Davide Jun 08 '19 at 15:15
@Davide you can't go by the checkin comments in the file itself, they haven't been updated in years, ever since Indy switched to SVN for source control. You are using the stock Indy that shipped with 10.3.1? Have you tried upgrading to the latest snapshot? – Remy Lebeau Jun 08 '19 at 16:59
I finally solved by rewriting in my app the OnCommandGet event code. Successully tested with many active clients uploading also huge files (6 GB!). @RemyLebeau thanks a lot. – Davide Jun 09 '19 at 08:43

score -1 · Answer 2 · answered Jun 07 '19 at 08:12

-1

Delphi by default seems to have some kind of limit on memory usage, adding this lines to .DPR project file:

const
    IMAGE_FILE_LARGE_ADDRESS_AWARE = $0020;

    {$SetPEFlags IMAGE_FILE_LARGE_ADDRESS_AWARE}

applications can use up to 2.5 GB of RAM on the 32bit versions of Windows and up to 3.5 GB of RAM on the 64bit versions. (https://cc.embarcadero.com/item/24309)

Anyway I think @RemyLebeau solution is the best

answered Jun 07 '19 at 08:12

Davide

39
4

Not only is this answer the wrong way to handle this issue, but it is also dangerous to enable the `IMAGE_FILE_LARGE_ADDRESS_AWARE` flag when your app is not *really* LargeAddress-aware, particularly in the RTL's memory manager. Besides, just because LAA would allow your app to access 2+ GB of RAM *total* still does not guarantee that the memory manager will be able to allocate 1+ GB *contiguously* for the `TMemoryStream` to use. – Remy Lebeau Jun 07 '19 at 19:45

TidHTTPServer "Out of memory" on large file upload

2 Answers2