How to determine the size of a buffer for a DLL call when the result comes from the DLL

Question

Using both Delphi 10.2 Tokyo and Delphi XE2.

I have a DLL that posts XML data to a site. The DLL is built with Delphi 10 in order to use TLS 1.2, which is not available with Delphi XE2.

The call to the DLL comes from a Delphi XE2 EXE, but I don't believe that is relevant, but I am noting it nonetheless.

The call to post data to a site will often return text data. Sometimes very large amounts of text data. Greater than 150K characters.

My original DLL convention was basically not correct, as I returned the contents of the returned text data as a PChar. In my readings here and elsewhere, that's a big no-no.

That "bad" methodology worked well until I started to get very large amounts of data returned. I tested it, and it failed on anything greater than 132,365 characters.

I restructured my DLL and calling code to pass in a buffer as a PChar to fill in, but I get an error trying to fill the output value!

Secondly, since I never know how big the returned data will be, how to I specify how big a buffer to fill from my calling method?

My DLL code where I get the error:

library TestDLL;

uses
  SysUtils,
  Classes,
  Windows,
  Messages,
  vcl.Dialogs,
  IdSSLOpenSSL, IdHTTP, IdIOHandlerStack, IdURI,
  IdCompressorZLib;

{$R *.res}

function PostAdminDataViaDll(body, method, url: PChar; OutData : PChar; OutLen : integer): integer; stdcall
var HTTPReq : TIdHTTP;
var Response: TStringStream;
var SendStream : TStringStream;
var IdSSLIOHandler : TIdSSLIOHandlerSocketOpenSSL;
var Uri : TIdURI;
var s : string;
begin
  Result := -1;
  try
    HTTPReq := TIdHTTP.Create(nil);
    IdSSLIOHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
    IdSSLIOHandler.SSLOptions.Mode := sslmClient;
    IdSSLIOHandler.SSLOptions.SSLVersions := [sslvTLSv1_2, sslvTLSv1_1];
    if Assigned(HTTPReq) then begin
      HTTPReq.Compressor := TIdCompressorZLib.Create(HTTPReq);
      HTTPReq.IOHandler := IdSSLIOHandler;
      HTTPReq.ReadTimeout := 180000;//set read timeout to 3 minutes
      HTTPReq.Request.ContentType := 'text/xml;charset=UTF-8';
      HTTPReq.Request.Accept := 'text/xml';
      HTTPReq.Request.CustomHeaders.AddValue('SOAPAction', 'http://tempuri.org/Administration/' + method);
      HTTPReq.HTTPOptions := [];
    end;
    SendStream := TStringStream.Create(Body);
    Response := TStringStream.Create(EmptyStr);
    try
      HTTPReq.Request.ContentLength := Length(Body);

      Uri := TiDUri.Create(url);
      try
        HTTPReq.Request.Host := Uri.Host;
      finally
        Uri.Free;
      end;

      HTTPReq.Post(url + 'admin.asmx', SendStream,Response);

      if Response.Size > 0 then begin
        if assigned(OutData) then begin
          s := Response.DataString;// Redundant? Probably can just use Response.DataString?
          StrPLCopy(OutData, s, OutLen);// <- ACCESS VIOLATION HERE
          //StrPLCopy(OutData, s, Response.Size);// <- ACCESS VIOLATION HERE
          Result := 0;
        end;
      end
      else begin
        Result := -2;
      end;
    finally
      Response.Free;
      SendStream.Free;
      IdSSLIOHandler.Free;
      HTTPReq.Free;
    end;
  except
    on E:Exception do begin
      ShowMessage(E.Message);
      Result := 1;
    end;
  end;
end;

exports
  PostAdminDataViaDll;

begin
end.

My Calling method code:

function PostAdminData(body, method, url : string): IXMLDOMDocument;
type
   TMyPost = function (body, method, url: PChar; OutData : PChar; OutLen : integer): integer; stdcall;
var Handle : THandle;
var MyPost : TMyPost;
var dataString : string;
var returnData : string;
begin
  if not (FileExists(ExtractFilePath(Application.ExeName) + 'TestDLL.DLL')) then begin
    Application.MessageBox(pchar('Unable to find TestDLL.DLL.'), pchar('Error posting'),MB_ICONERROR + MB_OK);
    Exit;
  end;

  dataString := EmptyStr;
  returnData := '';

  Handle := LoadLibrary(PChar(ExtractFilePath(Application.ExeName) + 'TestDLL.DLL'));
  if Handle <> 0 then begin
    try
      try
        MyPost := GetProcAddress(Handle, 'PostAdminDataViaDll');
        if @MyPost <> nil then begin
          // NOTE 32767 is not big enough for the returned data! Help!
          if MyPost(PChar(body), PChar(method), PChar(url), PChar(returnData), 32767) = 0 then begin
            dataString := returnData;
          end;
        end;
      except
      end;
    finally
      FreeLibrary(Handle);
    end;
  end
  else begin
    Application.MessageBox(pchar('Unable to find TestDLL.DLL.'), pchar('Error posting'),MB_ICONERROR + MB_OK);
  end;

  if not sametext(dataString, EmptyStr) then begin
    try
      Result := CreateOleObject('Microsoft.XMLDOM') as IXMLDOMDocument;
      Result.async := False;
      Result.loadXML(dataString);
    except
    end;
  end;
end;

Answer 1

I have a DLL that posts XML data to a site. The DLL is built with Delphi 10 in order to use TLS 1.2, which is not available with Delphi XE2.

Why not simply update Indy in XE2 to a newer version that supports TLS 1.2? Then you don't need the DLL at all.

My original DLL convention was basically not correct, as I returned the contents of the returned text data as a PChar. In my readings here and elsewhere, that's a big no-no.

It is not a "big no-no", especially if the response data is dynamic in nature. Returning a pointer to dynamically allocated data is perfectly fine. You would simply have to export an extra function to free the data when the caller is done using it, that's all. The "big no-no" is that this does introduce a potential memory leak, if the caller forgets to call the 2nd function. But that is what try..finally is good for.

That "bad" methodology worked well until I started to get very large amounts of data returned. I tested it, and it failed on anything greater than 132,365 characters.

That is not a lot of memory. Any failure you were getting with it was likely due to you simply misusing the memory.

I restructured my DLL and calling code to pass in a buffer as a PChar to fill in, but I get an error trying to fill the output value!

That is because you are not filling in the memory correctly.

Secondly, since I never know how big the returned data will be, how to I specify how big a buffer to fill from my calling method?

You can't, when using POST. You would have to cache the response data somewhere off to the side, and then expose ways to let the caller query that cache for its size and data afterwards.

My DLL code where I get the error:

My Calling method code:

I see a number of logic mistakes in that code.

But, the most important reason for the Access Violation error is that your EXE is simply not allocating any memory for its returnData variable.

Casting a string to a PChar never produces a nil pointer. If the input string is not empty, a pointer to the string's first Char is returned. Otherwise, a pointer to a static #0 Char is returned instead. This ensures that a string casted to PChar always results in a non-nil, null-terminated, C style character string.

Your EXE is telling the DLL that returnData can hold up to 32767 chars, but in reality it can't hold any chars at all! In the DLL, OutData is not nil, and OutLen is wrong.

Also, StrPLCopy() always null-terminates the output, but the MaxLen parameter does not include the null-terminator, so the caller must allocate room for MaxLen+1 characters. This is stated in the StrPLCopy() documentation.

With all of this said, try something more like this:

library TestDLL;

uses
  SysUtils,
  Classes,
  Windows,
  Messages,
  Vcl.Dialogs,
  IdIOHandlerStack, IdSSLOpenSSL, IdHTTP, IdCompressorZLib;

{$R *.res}

function PostAdminDataViaDll(body, method, url: PChar;
  var OutData : PChar): integer; stdcall;
var
  HTTPReq : TIdHTTP;
  SendStream : TStringStream;
  IdSSLIOHandler : TIdSSLIOHandlerSocketOpenSSL;
  s : string;
begin
  OutData := nil;

  try
    HTTPReq := TIdHTTP.Create(nil);
    try
      IdSSLIOHandler := TIdSSLIOHandlerSocketOpenSSL.Create(HTTPReq);
      IdSSLIOHandler.SSLOptions.Mode := sslmClient;
      IdSSLIOHandler.SSLOptions.SSLVersions := [sslvTLSv1, sslvTLSv1_1, sslvTLSv1_2];
      HTTPReq.IOHandler := IdSSLIOHandler;

      HTTPReq.Compressor := TIdCompressorZLib.Create(HTTPReq);
      HTTPReq.ReadTimeout := 180000;//set read timeout to 3 minutes
      HTTPReq.HTTPOptions := [];

      HTTPReq.Request.ContentType := 'text/xml';
      HTTPReq.Request.Charset := 'UTF-8';
      HTTPReq.Request.Accept := 'text/xml';
      HTTPReq.Request.CustomHeaders.AddValue('SOAPAction', 'http://tempuri.org/Administration/' + method);

      SendStream := TStringStream.Create(Body, TEncoding.UTF8);
      try
        s := HTTPReq.Post(string(url) + 'admin.asmx', SendStream);
      finally
        SendStream.Free;
      end;

      Result := Length(s);
      if Result > 0 then begin
        GetMem(OutData, (Result + 1) * Sizeof(Char));
        Move(PChar(s)^, OutData^, (Result + 1) * Sizeof(Char));
      end;
    finally
      HTTPReq.Free;
    end;
  except
    on E: Exception do begin
      ShowMessage(E.Message);
      Result := -1;
    end;
  end;
end;

function FreeDataViaDll(Data : Pointer): integer; stdcall;
begin
  try
    FreeMem(Data);
    Result := 0;
  except
    on E: Exception do begin
      ShowMessage(E.Message);
      Result := -1;
    end;
  end;
end;

exports
  PostAdminDataToCenPosViaDll,
  FreeDataViaDll;

begin
end.

function PostAdminData(body, method, url : string): IXMLDOMDocument;
type
   TMyPost = function (body, method, url: PChar; var OutData : PChar): integer; stdcall;
   TMyFree = function (Data  Pointer): integer; stdcall;
var
  hDll : THandle;
  MyPost : TMyPost;
  MyFree : TMyFree;
  dataString : string;
  returnData : PChar;
  returnLen : Integer;
begin
  hDll := LoadLibrary(PChar(ExtractFilePath(Application.ExeName) + 'TestDLL.DLL'));
  if hDll = 0 then begin
    Application.MessageBox('Unable to load TestDLL.DLL.', 'Error posting', MB_ICONERROR or MB_OK);
    Exit;
  end;
  try
    try
      MyPost := GetProcAddress(hDll, 'PostAdminDataViaDll');
      MyFree := GetProcAddress(hDll, 'FreeDataViaDll');
      if Assigned(MyPost) and Assigned(MyFree) then begin
        returnLen := MyPost(PChar(body), PChar(method), PChar(url), returnData);
        if returnLen > 0 then begin
          try
            SetString(dataString, returnData, returnLen);
          finally
            MyFree(returnData);
          end;
        end;
      end;
    finally
      FreeLibrary(hDll);
    end;
  except
  end;

  if dataString <> '' then begin
    try
      Result := CreateOleObject('Microsoft.XMLDOM') as IXMLDOMDocument;
      Result.async := False;
      Result.loadXML(dataString);
    except
    end;
  end;
end;