What could be causing IdTCPServer to fail before reading all the data OnExecute event?

Question

This code work's fine when I send data across the LAN with an Indy client component, but when I receive data from an external application from the web, it's causing it to fail. Could there be something on the client-side that is causing IdTCPServer to disconnect before all the data is read? An average of 33,000 characters are being sent by the client. Any suggestions?

procedure TFrmMain.IdTCPServer1Execute(AContext: TIdContext);
var
  strm: TMemoryStream;
  RxBuf: TIdBytes;
begin
  Memo1.Clear;
  strm := TMemoryStream.Create;
  try
    // read until disconnected
    AContext.Connection.IOHandler.ReadStream(strm, -1, true);
    strm.Position := 0;
    ReadTIdBytesFromStream(strm, RxBuf, strm.Size);
  finally
    strm.Free;
  end;
  Memo1.Lines.Add(BytesToString(RxBuf));
  AContext.Connection.IOHandler.WriteLn('000');
end;

I also tryed this other code, in this case unlike the first code it only reads part of the data beeing sent. Is there a way to make the IdTCPServer Handler wait until all the data is collected?

procedure TFrmMain.IdTCPServer1Execute(AContext: TIdContext);
var
  RxBuf: TIdBytes;
begin
  RxBuf := nil;
  with AContext.Connection.IOHandler do
  begin
    CheckForDataOnSource(10);
    if not InputBufferIsEmpty then
    begin
      InputBuffer.ExtractToBytes(RxBuf);
    end;
  end;
  AContext.Connection.IOHandler.WriteLn('000');
  Memo1.Lines.Add( BytesToString(RxBuf) );
end;

`...it's causing it to fail.` What does that mean? What did you observe that lead to the conclusion that this "failed"? — J..., Oct 19 '19 at 19:00
What happens if you use `AContext.Connection.IOHandler.ReadStream(strm, 0, true);`? — Freddie Bell, Oct 19 '19 at 19:36
The OnExecute event runs in a worker thread, not in the main UI thread. Your use of TMemo without synchronization is not thread safe. Also, if you are going to read all of the client's data until disconnected, there is no point in calling WriteLn afterwards since the client will never get it. And depending on the nature of the data, reading all of the data into one large memory buffer before using it may be too wasteful. What does the data actually look like? — Remy Lebeau, Oct 19 '19 at 21:58
When I say fail, I do a trace and when it's on this line: AContext.Connection.IOHandler.ReadStream(strm, -1, true); it jumps to the finally exception. — user734781, Oct 20 '19 at 13:08
Remy the data is an XML file type data of an average of 30,000 charactres.:
4017148526223433000152a237c8ce5f41c98f18b55f3d0f45c82019-10-17T23:02:08.0729819Z226613 — user734781, Oct 20 '19 at 13:17
@user734781 `...it jumps to the finally exception`. So... catch the exception and tell us what it is. — J..., Oct 20 '19 at 17:20
Remy, How can I loop to collect data until I get a specific character like eof? — user734781, Oct 21 '19 at 16:19

Remy Lebeau · Answer 1 · 2019-10-23T23:51:42.507

This code you posted as an answer is all wrong.

First off, you can't use BytesToString() on arbitrary byte blocks, that won't handle multi-byte encodings like UTF-8 correctly.

Also, you are not looking for the EOT terminator correctly. There is no guarantee that it will be the last byte of RxBuf after each read, if the client sends multiple XML messages. And even if it were, using Copy(BytesToString(), ...) to extract it into a string will never result in a blank string, like your code is expecting.

If the client sends an EOT terminator at the end of the XML, there is no need for a manual reading loop. Simply call TIdIOHandler.ReadLn() with the EOT terminator, and let it handle the read looping internally until the EOT is reached.

Also, the CoInitialize() and CoUninitialize() calls should be done in the OnConnect and OnDisconnect events, respectively (actually, they would be better called in a TIdThreadWithTask descendant assigned to the TIdSchedulerOfThread.ThreadClass property, but that is a more advanced topic for another time).

Try something more like this:

procedure TFrmMain.IdTCPServer1Connect(AContext: TIdContext);
begin
  CoInitialize(nil);
  AContext.Connection.IOHandler.DefStringEncoding := IndyTextEncoding_UTF8;
end;

procedure TFrmMain.IdTCPServer1Disconnect(AContext: TIdContext);
begin
  CoUninitialize();
end;

procedure TFrmMain.IdTCPServer1Execute(AContext: TIdContext);
var
  XML: string;
begin
  cdsSurescripts.Close;
  XML := AContext.Connection.IOHandler.ReadLn(#4);
  Display('CLIENT', XML);
  AContext.Connection.IOHandler.WriteLn('000');
end;

Personally, I would take a different approach. I would suggest using an XML parser that supports a push model. Then you can read arbitrary blocks of bytes from the connection and push them into the parser, letting it fire events to you for completed XML elements, until the terminator is reached. This way, you don't have to waste time and memory buffering the entire XML in memory before you can then process it.

score -1 · Answer 2 · answered Oct 22 '19 at 17:34

For further reference to anyone, I had to create a loop and wait for an EOT chr(4) send by the client in order to collect all the data on the IdTCPServer1Execute. This happens because the data is fragmented by Indy, The code looks something like this:

procedure TFrmMain.IdTCPServer1Execute(AContext: TIdContext);
var
  Len: Integer;
  Loop: Boolean;
begin
  CoInitialize(nil);
  cdsSurescripts.Close;
  Loop := True;
  while Loop = true do
  begin
    if AContext.Connection.IOHandler.Readable then
    begin
      AContext.Connection.IOHandler.ReadBytes( RxBuf,-1, True);
      Len := Length(BytesToString(RxBuf));
      if Copy(BytesToString(RxBuf), Len, 1) =  '' then
      begin
        loop := False;
      end;
    end;
  end;
  Display('CLIENT', BytesToString(RxBuf));
  AContext.Connection.IOHandler.WriteLn('000');
  CoUninitialize();
end;

This code is all wrong. See the answer I just posted. – Remy Lebeau Oct 23 '19 at 23:43 — Remy Lebeau, Oct 23 '19 at 23:43

What could be causing IdTCPServer to fail before reading all the data OnExecute event?

2 Answers2