How to correctly download csv without 'save as' dialog' using TWebbrowser and Delphi?

Question

I subscribe to a secure https web page containing a button that downloads some data as csv. I am trying to automate the download without the 'save as' dialog appearing but always seem to get an empty file downloaded. I suspect it has something to do with file type I'm using with IdHttp as most of my code works correctly. Please can anyone help with my use of IdHttp or see where else I am going wrong?

The download button on the site calls some javascript to perform the download as follows

<a class="dlCSV" href="javascript:void(0);"   onclick="dl_module.DownloadCsv();return false;">Download in CSV format…</a>

In Delphi I use a TWeb browser to log on securely and navigate to the page.

Clicking the download button in the TwebBrowser by hand shows the 'save as' dialog and then correctly downloads the csv data, defaulting to the filename 'data.csv'.

Automating clicking the button using execScript (below) also works, again showing the 'save as' dialog and correctly downloading the data with the same default filename.

procedure TForm1.BtnClickDownloadbuttonClick(Sender: TObject);
var  TheDocument : IHTMLDocument2;  // current HTML document
     HTMLWindow: IHTMLWindow2; // parent window of current HTML document
begin
TheDocument := WebBrowser1.Document as IHTMLDocument2; // Get reference to current document
if not Assigned(TheDocument) then
    Exit;

HTMLWindow := TheDocument.parentWindow;      // Get parent window of current document
if  Assigned(HTMLWindow) then
  try
    HTMLWindow.execScript('dl_module.DownloadCsv()', 'JavaScript'); // execute JS function to do download
  except  
  on E : Exception do
     begin
       showmessage ('Exception class name = '+E.ClassName+ slinebreak
                 +  'Exception message = '+E.Message);
      end  //on E
  end;
end;

Then I added TLama's code from here How do I keep an embedded browser from prompting where to save a downloaded file? to use IDownloadManager to intercept the download and prevent the 'save as' dialog. This is where it seems to go wrong as I then get an empty file downloaded, and not with the name data.csv.

My code for function TWebBrowser.Download, TWebBrowser.InvokeEvent, function TWebBrowser.QueryService and TForm1.FormCreate are identical to that provided by TLama in the link above.

My procedure TForm1.Button1Click is the same except that I changed the download function being called to the one on my page by changing the line

HTMLWindow.execScript('SRT_stocFund.Export()', 'JavaScript');

to

HTMLWindow.execScript('dl_module.DownloadCsv()', 'JavaScript');

and my procedure TForm1.BeforeFileDownload is identical except that because I'm on a secure site I added the variable

var
    LHandler: TIdSSLIOHandlerSocketOpenSSL; //<< on a https site    

and after creating the Filestream I added the lines

LHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil); 
IdHTTP.IOHandler := LHandler; 

The issue seems to be in procedure TForm1.BeforeFileDownload where I note that the value of FileSource is https://www.the_web_site_name/Ashx/GenericCSV.ashx.

There is a short delay while IdHTTP.Get(FileSource, FileStream); executes and then a file is created on my hard disc but called 'GenericCSV.ashx' (not data.csv) and the file is zero bytes long and completely empty.

Any ideas why its not downloading the file called data.csv (Do I somehow have to execute GenericCSV.ashx as well? if so how?)

For info here is my version of procedure TForm1.BeforeFileDownload

procedure TForm1.BeforeFileDownload(Sender: TObject;  const FileSource: WideString; var Allowed: Boolean);
var
  IdHTTP: TIdHTTP;
  FileTarget: string;
  FileStream: TMemoryStream;
  LHandler: TIdSSLIOHandlerSocketOpenSSL;  // added as its a https site
begin
  FileSourceEdit.Text := FileSource;
  Allowed := ShowDialogCheckBox.Checked;
  if not Allowed then
  try
    IdHTTP := TIdHTTP.Create(nil);
    try
      FileStream := TMemoryStream.Create;
      LHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil); //<<< added as its a https site
      IdHTTP.IOHandler := LHandler;    //<<< added as its a https site
      try
        IdHTTP.HandleRedirects := True;
        IdHTTP.Get(FileSource, FileStream);
        FileTarget := IdHTTP.URL.Document;
        if FileTarget = '' then
          FileTarget := 'File';
        FileTarget := ExtractFilePath(ParamStr(0)) + FileTarget;
        FileStream.SaveToFile(FileTarget);
      finally
        FileStream.Free;
      end;
    finally
      IdHTTP.Free;
    end;
    ShowMessage('Downloading finished! File has been saved as:' + sLineBreak +
      FileTarget);
  except
    on E: Exception do
      ShowMessage(E.Message);
  end;
end;

Answer 1

After you login, you can use this code to retrieve cookies from TWebBrowser

procedure GetHttpOnlyCookie(const AUrl: string; var ACookies: string);
const
  INTERNET_COOKIE_HTTPONLY = 8192;
var
  i: Integer;
  hModule: THandle;
  InternetGetCookieEx: function(lpszUrl, lpszCookieName, lpszCookieData
    : PAnsiChar; var lpdwSize: DWORD; dwFlags: DWORD; lpReserved: pointer)
    : BOOL; stdCall;
  CookieSize: DWORD;
  CookieData: PAnsiChar;
begin
  LoadLibrary('wininet.dll');
  hModule := GetModuleHandle('wininet.dll');
  if (hModule <> 0) then
  begin
    @InternetGetCookieEx := GetProcAddress(hModule, 'InternetGetCookieExA');
    if (@InternetGetCookieEx <> nil) then
    begin
      CookieSize := 1024;
      Cookiedata := AllocMem(CookieSize);
      try
        if InternetGetCookieEx(PAnsiChar(AUrl), nil, Cookiedata, CookieSize, INTERNET_COOKIE_HTTPONLY, nil) then
        begin
          ACookies:=CookieData;
        end;
      finally
        FreeMem(Cookiedata);
      end;
    end;
  end;
end;

Then you just parse your cookies and add them (you have to create CookieManager in IdHTTP first)

IdHTTP1.CookieManager.AddServerCookie();

Then you start your download and it should work if you passed all parameters correctly (unfortunately, it is not possible to find out what your site requires).

Answer 2

Thank you smooty86 but I think its time I gave up trying to doing it this way and simply parse the page I can see. I don't mind trying to understand code and adapting it to my needs but its so much harder trying to follow hints and suggestions when I'm working in the dark and especially don't know what parameters are needed everywhere. (I'm not daft, I've been programming for nearly 30 years and have spent over 4 years developing this particular data processing application but rarely touch web stuff)

However, the progress so far is...

Running your GetHttpOnlyCookie code after a successful login using automated filling in of the fields and clicking the login button returned an empty string so I used this code instead that at least seemed to return something that looked a little similar to your cookie string, ie seveveral strings separated by semicolons, most being name=value. (IdCookieManager1 is connected to IdHttp)

CookieList := Tstringlist.Create ;
try
  CookieList.Delimiter := ';' ;
  document := WebBrowser1.Document as IHTMLDocument2;
  CookieList.DelimitedText := document.cookie;
  for i := 0 to CookieList.Count-1 do
      IdCookieManager1.AddCookie(CookieList[i],LOGIN_URL)
finally
  CookieList.Free;
end;

Then in my original procedure BeforeFileDownload I try to log IdHttp into the site as well using code I adapted from here Log in to website from Delphi and the the cookies held in the cookie manager. Displaying the string returned showed lots of HTML that appeared to represent the oringinal log in page and not the page you see after log in

procedure TFrmInportGrades.BeforeFileDownload(Sender: TObject;  const FileSource: WideString; var Allowed: Boolean);
var
  FileTarget: string;
  FileStream: TMemoryStream;
  request : Tstringlist;
  s : string;
begin
  FileSourceEdit.Text := FileSource;
  Allowed := ShowDialogCheckBox.Checked;
  if not Allowed then
    begin
    try
    FileStream := TMemoryStream.Create;
    IdHTTP.CookieManager := IdCookieManager1;
    s := LogInIdHttp;  //<<<< log in the IdHttp
    showmessage(s); //<<<< debug
    IdHTTP.Get(FileSource, FileStream);
    FileTarget := IdHTTP.URL.Document;
    if FileTarget = '' then
          FileTarget := 'File';
    FileTarget := ExtractFilePath(ParamStr(0)) + FileTarget;
    FileStream.SaveToFile(FileTarget);
    finally
        FileStream.Free;
    end;

    ShowMessage('Downloading finished! File has been saved as:' + sLineBreak +
      FileTarget);
    end;
end;

The login code I used is below but I don't really know what I am doing here or what needs to be put into the Request.Add() parameters. I used 'Inspect element' from firefox to get the name of the user and password boxes and put the correct users name and password after the '=' sign in lines {3} and {4}. In lines {2},{6} and {7} I put the url of the log in site. I've no idea what lines {1}, {2}, {5} do or even if they are correct or necessary

function TFrmInportGrades.LogInIdHttp: string;
var
  Request: TStringList;
  Response: TMemoryStream;
  LHandler: TIdSSLIOHandlerSocketOpenSSL;  // added as its a https site
begin
   Result := '';
   try
     Response := TMemoryStream.Create;
     try
       Request := TStringList.Create;
        try
    {1}    Request.Add('op=login');
    {2}    Request.Add('redirect=https://www.thewebsite.com/Login.aspx' );
    {3}    Request.Add('ctl00$ctl00$Body$Body$loginManager$ctl00$loginEmailInput=usernme');
    {4}    Request.Add('ctl00$ctl00$Body$Body$loginManager$ctl01$passwordInput=password'});
           LHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil); //<<< added as its a https site
           IdHTTP.IOHandler := LHandler;    //<<< added as its a https site
           IdHTTP.AllowCookies := True;
           IdHTTP.HandleRedirects := True;
     {5}   IdHTTP.Request.ContentType := 'application/x-www-form-urlencoded';
     {6}   IdHTTP.Post('https://www.thewebsite.com/Login.aspx', Request, Response);
     {7}    Result := IdHTTP.Get('https://www.thewebsite.com/Login.aspx');
       finally
        Request.Free;
      end;
    finally
      Response.Free;
    end;
  except
    on E: Exception do
      ShowMessage(E.Message);
  end;
end;

The net result of all this is that I don't get a file created at all now, not even a zero byte one. This all seems very overcomplicated simply to avoid or automate the 'Save As' dialog and is requiring lots of code that I won't be able to maintan afterwards. Unless somebody has a simpler solution I'll just parse what I can see (BTW I tried TEmbeddedWebBrowser but there is so little documentation for it I couldn't see how to make it download correctly. Might try again later.) Thank you for trying to help!