Delphi Indy JSON Post

Question

So I have been trying to get the data I require from a website using the Indy Project (Internet Direct) in Delphi. I have managed to use the GET and POST methods to log in, obtain the unique Session ID (EPASS_Nonce_ID) that is required to be appended to all URLs once logged in, and now I am trying to extract the data. I think that the page is using JSON, however I have not used this before.

I was able to use the GET method to obtain the source code, but it was devoid of the content (the content I wanted anyways). I think this is because the JSON script has to run to load the data, and my simple GET method doesn't invoke this.

It seems that there is a script running in the background to obtain the CSV data that is used to fill a table on the Website. The JSON appears to provide a request to the server and displays "Processing" on the UI while that is waiting. Once it has the response, it seems to change the UI to "My Transaction History" and display the data.

While I can use GET to obtain the website, I need assistance in initiating the JSON script (or whatever I need to do) in order to obtain the CSV data.

HTML JSON Script:

<script type="application/json" id="jsparams">{"loadAction":"/website/transaction_history2/_md_transaction_history2_r1_content/view/_ac_transaction_history2_r1_content/AC/_pid
/transaction_history2_r1_content/_st_transaction_history2_r1_content/normal?action\u003dloadAction\u0026EPASS_NONCE_ID\u003dE20F7163FA571F7FB19FD92F1AE997A6","lastLogin"
:1513917404587,"lastLoginDateFormat":"E NNN d yyyy, HH:mm (Ti\\m\\e Zon\\e z G\\MT)","externalPageUrl":"/website/external_page.jsp?EPASS_NONCE_ID\u003dE20F7163FA571F7FB19FD92F1AE997A6\u0026link\
u003d","contextPath":"/comsuper_mil","windowTitle":"My Transaction History","csvAction":"/website/transaction_history2/_md_transaction_history2_r1_content/view/_ac_transaction_history2_r1_content/AC/_pid/transaction_history2_r1_content/
_st_transaction_history2_r1_content/normal?action\u003dcsvAction\u0026EPASS_NONCE_ID\u003dE20F7163FA571F7FB19FD92F1AE997A6"}</script>

Here is the entire response I receive from a Get or Post method used on the site:

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en" >
<head>
    <title>ePASS</title>
    <meta name="keywords" content="ePASS" />
    <meta name="description" content="ePASS Member Module" />
    <meta name="author" content="Bravura Solutions Limited" />
    <meta http-equiv="content-type" content="text/html; charset=iso-8859-1" />
    <!--  IE should be as standards compatible as possible -->
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <link rel="stylesheet" media="screen" type="text/css" title="standard" href="/super_mil/common/stylesheets/core.css"/>
    <link rel="stylesheet" media="screen" type="text/css" title="standard" href="/super_mil/common/stylesheets/bpay.css"/>
    <link rel="stylesheet" media="screen" type="text/css" title="standard" href="/super_mil/common/stylesheets/global.css"/>
    <link rel="stylesheet" media="print" type="text/css" title="standard" href="/super_mil/common/stylesheets/print.css"/>
    <link rel="stylesheet" media="screen" type="text/css" title="standard" href="/super_mil/member/jepass.css"/>
    <link rel="stylesheet" media="screen" type="text/css" title="standard" href="/super_mil/member/msbs/jepass.css"/>
    <link rel="stylesheet" media="print" type="text/css" title="standard" href="/super_mil/member/msbs/print.css"/>
    <script type="text/javascript" src="/super_mil/common/javascript/jquery.min.js"></script>
    <!-- jquery ui-->
    <link type="text/css" href="/super_mil/common/javascript/jquery/themes/base/jquery.ui.all.css" rel="Stylesheet" />
    <script type="text/javascript" src="/super_mil/common/javascript/jquery/js/jquery-ui.min.js"></script>
    <script type="text/javascript" src="/super_mil/common/javascript/global.js"></script>
    <script type="text/javascript" src="/super_mil/common/javascript/client.js"></script>
    <script type="text/javascript" src="/super_mil/common/javascript/multi_select.js"></script>
    <script type="text/javascript" src="/super_mil/common/javascript/block_iframe.js"></script>
    <script type="text/javascript" src="/super_mil/common/javascript/jquery    /js/jquery-ui.min.js"></script>
    <link rel="stylesheet" href="/super_mil/common/javascript/jquery/themes/base/jquery.ui.all.css">
    <script type="text/javascript" src="/super_mil/common/javascript/datepicker.js"></script>
</head>
<body class="with-menu">
    <div id="t-wrapper">
    <!-- BEGIN Portlet -->
        <div id="page-wrapper">
        <div id="t-logo">
        <div id="t-sitelogo" ></div>
        <!-- Note: template.header -->
        <div id="l-template-header" class="header-note">
        <div id="top-spacer">    <div id="header_topstrip"> 
        <a shape="rect" class="topstrip_link" target="_ext" href="http://website/contact"><img src="/super_mil/common/images/contact_us.png" /></a> 
        </div>   </div>
        <div id="scheme_block"> <div id="scheme_img">
        <a shape="rect" class="topstrip_link" target="_ext" href="http://website"><img src="/super_mil/common/images/MSBS_scheme_block.png" /></a> 
        </div> </div>
        <div id='menu_header'>Member Services</div>
        </div>
    <div id="t-sitelogo-right" ></div>
    </div>
<!-- END Portlet --><!-- BEGIN Portlet -->
<div id="t-info">
<p class="info-text">
<script type="text/javascript" src="/super_mil/common/javascript/set_last_login_date.js"></script>  
        <span id="greeting_message">Good Afternoon</span> <span id="name">Person Name</span> <strong>Your Account Number is: <span id="text_2" class="readonly text">Account</span> (<span id="text_8" class="readonly text">Contributor</span>)</strong>
</p>
</div>
<!-- END Portlet --><!-- BEGIN Portlet -->
    <div class="main-content">
        <div class="middle-bit">
<!-- END Portlet --><!-- BEGIN Portlet -->
<script type="text/javascript" src="/super_mil/common/javascript/external_page.js"></script> 
<div id="tool-menu">
    <ul>
<li><a id="tool_print" href="javascript:print()" target="_self" tabindex="1">Print</a></li>
<li><a id="tool_contact" href="/super_mil/members/contact?EPASS_NONCE_ID=DA9CA80B99F7101EF6A2CD0B56245FEF" target="_self" tabindex="2">Contact Us</a></li>
<li><a id="tool_logout" href="/super_mil/members/transaction_history2/_ps_transaction_history2_r1_content/normal/_ac_transaction_history2_r1_menu/AC/_rp_transaction_history2_r1_content_transaction0x1type0x1filter/1_All/_rp_transaction_history2_r1_content_loadonchange/1_false/_md_transaction_history2_r1_content/view/_pm_transaction_history2_r1_content/view/_pid/transaction_history2_r1_menu/_rp_transaction_history2_r1_content_action/1_loadAction/_rp_transaction_history2_r1_content_date0x1filter/1_10x370x32017/_st_transaction_history2_r1_content/normal?action=logout&amp;EPASS_NONCE_ID=DA9CA80B99F7101EF6A2CD0B56245FEF" target="_self" tabindex="3">Log Off</a></li>
</ul>
</div>
<div id="t-mainmenu">
    <ul class="top-menu">
                <li class="menu-item menu-selected "><a class="menu-selected " href="/super_mil/members/transaction_history2/_ps_transaction_history2_r1_content/normal/_ac_transaction_history2_r1_menu/AC/_rp_transaction_history2_r1_content_transaction0x1type0x1filter/1_All/_rp_transaction_history2_r1_content_loadonchange/1_false/_md_transaction_history2_r1_content/view/_pm_transaction_history2_r1_content/view/_pid/transaction_history2_r1_menu/_rp_transaction_history2_r1_content_action/1_loadAction/_rp_transaction_history2_r1_content_date0x1filter/1_10x370x32017/_st_transaction_history2_r1_content/normal?action=menu&menu_id=1&EPASS_NONCE_ID=DA9CA80B99F7101EF6A2CD0B56245FEF&expand=maintenance" target="_self" >My Details</a>
            </li>
                <li class="menu-item "><a href="/super_mil/members/transaction_history2/_ps_transaction_history2_r1_content/normal/_ac_transaction_history2_r1_menu/AC/_rp_transaction_history2_r1_content_transaction0x1type0x1filter/1_All/_rp_transaction_history2_r1_content_loadonchange/1_false/_md_transaction_history2_r1_content/view/_pm_transaction_history2_r1_content/view/_pid/transaction_history2_r1_menu/_rp_transaction_history2_r1_content_action/1_loadAction/_rp_transaction_history2_r1_content_date0x1filter/1_10x370x32017/_st_transaction_history2_r1_content/normal?action=menu&menu_id=222&EPASS_NONCE_ID=DA9CA80B99F7101EF6A2CD0B56245FEF&expand=transaction_history2" class="" target="_self" >My Transaction History</a>
            </li>
                <li class="menu-item has-sub-menu"><a href="/super_mil/members/transaction_history2/_ps_transaction_history2_r1_content/normal/_ac_transaction_history2_r1_menu/AC/_rp_transaction_history2_r1_content_transaction0x1type0x1filter/1_All/_rp_transaction_history2_r1_content_loadonchange/1_false/_md_transaction_history2_r1_content/view/_pm_transaction_history2_r1_content/view/_pid/transaction_history2_r1_menu/_rp_transaction_history2_r1_content_action/1_loadAction/_rp_transaction_history2_r1_content_date0x1filter/1_10x370x32017/_st_transaction_history2_r1_content/normal?action=menu&menu_id=107&EPASS_NONCE_ID=DA9CA80B99F7101EF6A2CD0B56245FEF&expand=investment_change_2" class="" target="_self" >My Investments</a>
                            <ul class="sub-menu inactive">
                   <li class="sub-menu-item"><a href="/super_mil/members/transaction_history2/_ps_transaction_history2_r1_content/normal/_ac_transaction_history2_r1_menu/AC/_rp_transaction_history2_r1_content_transaction0x1type0x1filter/1_All/_rp_transaction_history2_r1_content_loadonchange/1_false/_md_transaction_history2_r1_content/view/_pm_transaction_history2_r1_content/view/_pid/transaction_history2_r1_menu/_rp_transaction_history2_r1_content_action/1_loadAction/_rp_transaction_history2_r1_content_date0x1filter/1_10x370x32017/_st_transaction_history2_r1_content/normal?action=menu&menu_id=1001&EPASS_NONCE_ID=DA9CA80B99F7101EF6A2CD0B56245FEF&expand=investment_change_2_sub_menu" class="" >View / Switch My Investment Details</a></li>
                </ul>
            </li>
                <li class="menu-item "><a href="/super_mil/members/transaction_history2/_ps_transaction_history2_r1_content/normal/_ac_transaction_history2_r1_menu/AC/_rp_transaction_history2_r1_content_transaction0x1type0x1filter/1_All/_rp_transaction_history2_r1_content_loadonchange/1_false/_md_transaction_history2_r1_content/view/_pm_transaction_history2_r1_content/view/_pid/transaction_history2_r1_menu/_rp_transaction_history2_r1_content_action/1_loadAction/_rp_transaction_history2_r1_content_date0x1filter/1_10x370x32017/_st_transaction_history2_r1_content/normal?action=menu&menu_id=17&EPASS_NONCE_ID=DA9CA80B99F7101EF6A2CD0B56245FEF&expand=client_page3" class="" target="_self" >My Statements</a>
            </li>
                <li class="menu-item "><a href="/super_mil/members/transaction_history2/_ps_transaction_history2_r1_content/normal/_ac_transaction_history2_r1_menu/AC/_rp_transaction_history2_r1_content_transaction0x1type0x1filter/1_All/_rp_transaction_history2_r1_content_loadonchange/1_false/_md_transaction_history2_r1_content/view/_pm_transaction_history2_r1_content/view/_pid/transaction_history2_r1_menu/_rp_transaction_history2_r1_content_action/1_loadAction/_rp_transaction_history2_r1_content_date0x1filter/1_10x370x32017/_st_transaction_history2_r1_content/normal?action=menu&menu_id=19&EPASS_NONCE_ID=DA9CA80B99F7101EF6A2CD0B56245FEF&expand=client_page5" class="" target="_self" >Reported Values and Caps</a>
            </li>
                <li class="menu-item "><a href="/super_mil/members/transaction_history2/_ps_transaction_history2_r1_content/normal/_ac_transaction_history2_r1_menu/AC/_rp_transaction_history2_r1_content_transaction0x1type0x1filter/1_All/_rp_transaction_history2_r1_content_loadonchange/1_false/_md_transaction_history2_r1_content/view/_pm_transaction_history2_r1_content/view/_pid/transaction_history2_r1_menu/_rp_transaction_history2_r1_content_action/1_loadAction/_rp_transaction_history2_r1_content_date0x1filter/1_10x370x32017/_st_transaction_history2_r1_content/normal?action=menu&menu_id=16&EPASS_NONCE_ID=DA9CA80B99F7101EF6A2CD0B56245FEF&expand=client_page2" class="" target="_self" >i-Estimator</a>
            </li>
                <li class="menu-item "><a href="/super_mil/members/transaction_history2/_ps_transaction_history2_r1_content/normal/_ac_transaction_history2_r1_menu/AC/_rp_transaction_history2_r1_content_transaction0x1type0x1filter/1_All/_rp_transaction_history2_r1_content_loadonchange/1_false/_md_transaction_history2_r1_content/view/_pm_transaction_history2_r1_content/view/_pid/transaction_history2_r1_menu/_rp_transaction_history2_r1_content_action/1_loadAction/_rp_transaction_history2_r1_content_date0x1filter/1_10x370x32017/_st_transaction_history2_r1_content/normal?action=menu&menu_id=9&EPASS_NONCE_ID=DA9CA80B99F7101EF6A2CD0B56245FEF&expand=contact" class="" target="_self" >Contact Us</a>
            </li>
                <li class="menu-item "><a href="/super_mil/members/transaction_history2/_ps_transaction_history2_r1_content/normal/_ac_transaction_history2_r1_menu/AC/_rp_transaction_history2_r1_content_transaction0x1type0x1filter/1_All/_rp_transaction_history2_r1_content_loadonchange/1_false/_md_transaction_history2_r1_content/view/_pm_transaction_history2_r1_content/view/_pid/transaction_history2_r1_menu/_rp_transaction_history2_r1_content_action/1_loadAction/_rp_transaction_history2_r1_content_date0x1filter/1_10x370x32017/_st_transaction_history2_r1_content/normal?action=menu&menu_id=15&EPASS_NONCE_ID=DA9CA80B99F7101EF6A2CD0B56245FEF&expand=client_page1" class="" target="_self" >My Forms</a>
            </li>
                <li class="menu-item "><a href="/super_mil/members/transaction_history2/_ps_transaction_history2_r1_content/normal/_ac_transaction_history2_r1_menu/AC/_rp_transaction_history2_r1_content_transaction0x1type0x1filter/1_All/_rp_transaction_history2_r1_content_loadonchange/1_false/_md_transaction_history2_r1_content/view/_pm_transaction_history2_r1_content/view/_pid/transaction_history2_r1_menu/_rp_transaction_history2_r1_content_action/1_loadAction/_rp_transaction_history2_r1_content_date0x1filter/1_10x370x32017/_st_transaction_history2_r1_content/normal?action=logout&EPASS_NONCE_ID=DA9CA80B99F7101EF6A2CD0B56245FEF&expand=logoff" class="" target="_self" >Log Off</a>
            </li>
    </ul>
</div>
<!-- END Portlet --><!-- BEGIN Portlet -->
<script type="text/javascript">
    top.document.title = 'Window Title';
</script>
<title>Window Title</title>
<script type="text/javascript" src="/super_mil/common/javascript/monitor_f.js"> </script>
    <script type="text/javascript" src="/super_mil/common/javascript/set_window_title.js"></script>
    <div id="l-main" class="l-monitor"> 
    <div id="l-monitor">
        <div id="print-logo" class="print-only">
            <img src="/super_mil/member/epass/images/print_logo.gif" alt="null" />
        </div>
                    <div id="l-title">
                <h1 class="title-text">My Transaction History</h1>
            </div>
            <!-- Note: monitor.header -->
<div id="l-header" class="header-note">
</div>
    <div id="l-monitor">
        <div id="l-actionarea">
            <br />
            <div id="l-message">
                <span class="message">Processing<span>
            </div>
            <br />
            <table class="results-table" >
                    <tr class="even row-1">
                        <td class="item column_null">&nbsp;</td>
                        <td class="item column_null"><img id="status_0" src="/super_mil/member/epass/icons/progress_in_progress.gif" /></td>
                    </tr>
            </table>
        </div>
        <div id="summary"></div>
        <br /><br />
        <div id="l-error">
            <ul>
                <li><span id="error" >An error occurred trying to process your request.</span></li>
            </ul>
        </div>
        <div class="buttons">
    <span class="button-span hidden_button" ><a id="ok_button" class="hidden_button" href="/super_mil/?EPASS_NONCE_ID=DA9CA80B99F7101EF6A2CD0B56245FEF" target="_self" tabindex="1"><span>OK</span></a></span>
&nbsp;<span class="button-span hidden_button" ><a id="yes_button" class="hidden_button" href="/super_mil/?EPASS_NONCE_ID=DA9CA80B99F7101EF6A2CD0B56245FEF" target="_self" tabindex="2"><span>Yes</span></a></span>
&nbsp;<span class="button-span hidden_button" ><a id="no_button" class="hidden_button" href="/super_mil/?EPASS_NONCE_ID=DA9CA80B99F7101EF6A2CD0B56245FEF" target="_self" tabindex="3"><span>No</span></a></span>
&nbsp;<span class="button-span hidden_button" ><a id="cancel_button" class="hidden_button" href="/super_mil/?EPASS_NONCE_ID=DA9CA80B99F7101EF6A2CD0B56245FEF" target="_self" tabindex="4"><span>Cancel</span></a></span>
&nbsp;<span class="button-span hidden_button" ><a id="download_button" class="hidden_button" href="/super_mil/?EPASS_NONCE_ID=DA9CA80B99F7101EF6A2CD0B56245FEF" target="_self" tabindex="5"><span>Download</span></a></span>
</div>
    </div>  
    <!-- Note: monitor.footer -->
<div id="l-footer" class="footer-note">
</div>
    </div>
    <div class="min-height-clear"></div>
<!-- main-content-middle -->
</div> <!-- end of l-main div -->
<!-- END Portlet --><!-- BEGIN Portlet -->
<!-- END Portlet --><!-- BEGIN Portlet -->
    <!-- main-content-footer -->
                <div class="min-height-clear"></div>
            </div><!-- end middle-bit -->
        </div><!-- end main-content -->
        </div><!-- end page-wrapper -->
        <div id="table-footer">
            <div id="footer">
                <!-- Note: template.footer -->
<div id="l-template-footer" class="footer-note">
<div class="footer">  <div id="footer-middle"> <p>
<a shape="rect" class="topstrip_link" target="_ext" href="http://website/privacy">Privacy Policy</a>  | 
<a shape="rect" class="topstrip_link" target="_ext" href="http://website/disclaimer">Disclaimer</a>
</p>
</div>  <div id="footer-bottom">
<b> Superannuation Company (SC)</b> Business Numbers <br />Trustee of the Superannuation and Benefits Scheme </div></div>
</div>
            </div>
        </div>  
<!-- END Portlet --><script type="text/javascript" src="/super_mil/common/javascript/set_first_focus.js"></script>
    </div>
<script type="application/json" id="jsparams">{"pollerUrl":"/super_mil/poller?EPASS_NONCE_ID\u003dDA9CA80B99F7101EF6A2CD0B56245FEF","lastLogin":1514254869743,"lastLoginDateFormat":"E NNN d yyyy, HH:mm (Ti\\m\\e Zon\\e z G\\MT)","externalPageUrl":"/super_mil/member/external_page.jsp?EPASS_NONCE_ID\u003dDA9CA80B99F7101EF6A2CD0B56245FEF\u0026link\u003d","resourcePath":"/super_mil/member/epass/","contextPath":"/super_mil","windowTitle":"Processing","actionName":"monitoredAction.au.com.essential.epass.member.portlet.TransactionHistory2Portlet.1731515MSBS"}</script>
</body>
</html>

And Using Fiddler, it seems that there are 3 separate GET requests made using/returning JSON data.

The first is:

actionName=monitoredAction.au.com.essential.epass.member.portlet.TransactionHistory2Portlet.1731515MSBS
contextPath=/comsuper_mil
externalPageUrl=/super_mil/member/external_page.jsp?EPASS_NONCE_ID=8E4F382666E719688D23B72BBD40D71E&link=
lastLogin=1514254720197
lastLoginDateFormat=E NNN d yyyy, HH:mm (Ti\m\e Zon\e z G\MT)
pollerUrl=/super_mil/poller?EPASS_NONCE_ID=8E4F382666E719688D23B72BBD40D71E
resourcePath=/super_mil/member/epass/
windowTitle=Processing

Then the second is:

choices
    {}
        choice=AUTO
        url=/super_mil/members/transaction_history2/_ac_transaction_history2_r1_content/AC/_pid/transaction_history2_r1_content?action=progressCompletedAction&EPASS_NONCE_ID=8E4F382666E719688D23B72BBD40D71E
stagesCompleted=0

Then the final one is:

contextPath=/super_mil
csvAction=/super_mil/members/transaction_history2/_md_transaction_history2_r1_content/view/_ac_transaction_history2_r1_content/AC/_pid/transaction_history2_r1_content/_st_transaction_history2_r1_content/normal?action=csvAction&EPASS_NONCE_ID=8E4F382666E719688D23B72BBD40D71E
externalPageUrl=/super_mil/member/external_page.jsp?EPASS_NONCE_ID=8E4F382666E719688D23B72BBD40D71E&link=
lastLogin=1514254720197
lastLoginDateFormat=E NNN d yyyy, HH:mm (Ti\m\e Zon\e z G\MT)
loadAction=/super_mil/members/transaction_history2/_md_transaction_history2_r1_content/view/_ac_transaction_history2_r1_content/AC/_pid/transaction_history2_r1_content/_st_transaction_history2_r1_content/normal?action=loadAction&EPASS_NONCE_ID=8E4F382666E719688D23B72BBD40D71E
windowTitle=My Transaction History

Try using [fiddler](https://www.telerik.com/fiddler) to capture all the requests/responses that are happening and then try to find which one(s) have the data you want and then try to replicate the request(s) that you need to get that data. — garbb, Dec 23 '17 at 22:35
@garbb thanks. I’ve been using Ethereal, but I can’t locate anything that looks like a JSON specific request or response. That is how I’ve done my GET and POST methods up until this point. — Sercho, Dec 23 '17 at 22:37
The "script" you show is not an actual script at all. It is just JSON wrapped in a ` — Remy Lebeau, Dec 24 '17 at 02:52
@RemyLebeau I triedto do what you stated, however I could not "act on it accordingly". Is there any chance you could provide a clue as to how to initiate the JSON request to the server? I have included significantly more detail in the edited question, which hopefully doesn't confuse you any more than it did me. Thanks again for your help. — Sercho, Dec 26 '17 at 03:39
@Sercho you are just throwing around random data at us, but no details, no explanations of what it is you are actually looking for. But it looks like the last piece of data you showed has a URL to CSV data that you are probably interested in (`csvAction=/super_mil/members/transaction_history2/_md_transaction_history2_r1_content/view/_ac_transaction_history2_r1_content/AC/_pid/transaction_history2_r1_content/_st_transaction_history2_r1_content/normal?action=csvAction&EPASS_NONCE_ID=8E4F382666E719688D23B72BBD40D71E`) — Remy Lebeau, Dec 26 '17 at 16:29
@RemyLebeau the problem is I really have no idea where to start with JSON. Yes the data I want is the CSV data, as i explained in my edited question (which I only discovered after looking into your first suggestion). I have gotten through using GET and POST methods, but I’m now stuck because it seems when I get the page, it is before the CSV data is loaded. Are you saying that I should be able to use a GET method to obtain the CSV data directly from that address? — Sercho, Dec 26 '17 at 18:46
@Sercho you tell us. You are the one with access to the server, not us. What happens if you `GET` that URL? Does it have the CSV data you want? The HTML table has to get the CSV from somewhere, you have to figure out what tells it where that somewhere is. This is where an HTTP debugger comes in handy. Fiddler, or even the debugger built in to your web browser. You are focused too much on JSON and not enough on HTTP/HTML basics — Remy Lebeau, Dec 26 '17 at 19:53
@RemyLebeau ok, I understand the point. I will play around some more. I have been using Fiddler, but I can’t see any direct GET methods on the CSV data. Thanks for all the help, if nothing else,you have assisted in eliminating several options. — Sercho, Dec 26 '17 at 19:56
@Sercho did you check for other request types besides `GET`? Did you try your browser's own request debugger? Fiddler might not catch everything — Remy Lebeau, Dec 26 '17 at 22:27
@RemyLebeau sorry for the delay. I tried the GET on the csvAction URL, but got a 403 Error. I have used Ethereal, Fiddler, Mozilla debugger and found nothing but JavaScripts and GET (once the transaction page is loaded). I tried to submit both Forms in the HTML using POST (even though no POST was tracked with sniffers)...which only results in the HTML being returned without the CSV data in the results. I have found references to a JQuery in the FireFox debugger, so I will do some more reading on Java and see whether there is a way for me to actually run the scripts through the server. Thanks! — Sercho, Dec 28 '17 at 21:14
@RemyLebeau thanks for all the help. See below for how i 'solved' this — Sercho, Dec 30 '17 at 05:15

score 0 · Answer 1 · answered Dec 30 '17 at 05:14

As it turned out...I found a very simple answer that I hadn't thought of trying.

The page I was trying to download the CSV data from, automatically loaded the current year's data onto the page using a Java Script query back to the server. While I couldn't figure out how to decode that query, or to simulate it...I noticed that I could change the year I was looking at. I used Fiddler to capture the packets again this time changing to last Financial Year, and then back to the current year (after it loaded). This generated discrete POST Methods to the server, which I was easily able to replicate, using:

Params2 := TStringList.Create;
Params2.Add('loadonchange=false');
Params2.Add('transaction_type_filter=All');
Params2.Add('date_filter=1/7/' + IntToStr(Last_S_Year));
Response := idhttp.Post(website + '/members/transaction_history2' + '/_md_transaction_history2_r1_content/view/' + '_ac_transaction_history2_r1_content/AC/_ps_transaction_history2_r1_content/normal/_pm_transaction_history2_r1_content/view/_pid/transaction_history2_r1_content/_st_transaction_history2_r1_content/normal?action=csvAction&' + EPASS,Params2);

This code resulted in me obtaining the exact CSV data in the Response string.

I know this doesn't strictly answer my question that I had...but it might outline a method for finding a unique way to obtain the Post or Get method you are trying to obtain from your site.

Delphi Indy JSON Post

1 Answers1