httpclient delivering different results

Question

I am having issues interfacing with a PowerSchool system using Apache HttpClient 4.5.6. I have left out methods that aren't necessary for this question, but I am 100% sure that they work as intended.

When using Postman or requests.py, using the exact same form data (I've verified it across python, java, and Firefox), I get the intended HTML page with the desired grades and class information in between a  and  comment (over 1000 lines). However, when using the same form data and headers in Java, the resulting HTML page contains only this:

<!-- start student content -->
<div id="quickLookup">

<tr>
<th class="right" colspan="19">Attendance Totals</th>
<th>0</th>
<th>0</th>
</table>
<table border="0" cellpadding="3" cellspacing="1" width="100%">


<tr>
<td align="center">Current Cumulative GPA (Q1): X.XXXX</td>
</tr>

<tr>
<td align="center"><a href="home.html?schoolid=XXXX&showdropped=true&91146885685933636948">Show dropped classes also</a></td>
</tr>
</table>

<tr>
<th class="right" colspan="10">Attendance Totals</th>
<th>0</th>
<th>0</th>
<th>0</th>
<th>0</th>
</table>

</div>
<!-- end student content -->

The disconnect between the two different results makes no sense to me, since as far as I know Postman and requests.py don't execute javascript. I would expect HttpClient's result to be the same. Here is my code:

private static final BasicCookieStore cookieStore = new BasicCookieStore();
private static final HttpClient client = HttpClientBuilder.create().setDefaultCookieStore(cookieStore).build();

public static void main(String[] args) {

        String baseUrl = "https://powerschoolinstallurl/";
        String username = "username";
        String password = "password";

        try {
            // get hidden data fields, calc hmac data
            HashMap<String, String> result = getAuthCodes(baseUrl);
            String dbpwField = getDBPWField(result.get("contextData"), password);
            String pwField = getPWField(result.get("contextData"), password);

            List<NameValuePair> form = new ArrayList<>();
            form.add(new BasicNameValuePair("pstoken", result.get("pstoken")));
            form.add(new BasicNameValuePair("contextData", result.get("pstoken")));
            form.add(new BasicNameValuePair("dbpw", dbpwField));
            form.add(new BasicNameValuePair("serviceName", "PS Parent Portal"));
            form.add(new BasicNameValuePair("pcasServerURL", "/"));
            form.add(new BasicNameValuePair("credentialType", "User Id and Password Credential"));
            form.add(new BasicNameValuePair("account", username));
            form.add(new BasicNameValuePair("pw", pwField));
            form.add(new BasicNameValuePair("ldappassword", password));

            UrlEncodedFormEntity requestEntity = new UrlEncodedFormEntity(form);

            HttpPost postMethod = new HttpPost(baseUrl + "guardian/home.html");
            postMethod.setEntity(requestEntity);

            HttpResponse rawResponse = client.execute(postMethod);
                    System.out.println(rawResponse.getStatusLine().getStatusCode());
            try {
                String responseString = new BasicResponseHandler().handleResponse(rawResponse);
                System.out.println(responseString);
            } catch (HttpResponseException ignore) {}
            System.out.println(cookieStore.getCookies().toString());

            HttpGet getMethod = new HttpGet(baseUrl + "guardian/home.html");
            // replicating headers, result is the same nontheless
            getMethod.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
            getMethod.setHeader("Accept-Encoding", "gzip, deflate, br");
            getMethod.setHeader("Accept-Language", "en-US,en;q=0.5");
            getMethod.setHeader("Referer", "https://powerschoolinstallurl/public/home.html");
            getMethod.setHeader("DNT", "1");
            getMethod.setHeader("Connection", "keep-alive");
            getMethod.setHeader("Cache-Control", "no-cache");
            getMethod.setHeader("Host", "ps.install.domain");
            getMethod.setHeader("Upgrade-Insecure-Requests", "1");
            getMethod.setHeader("Pragma", "no-cache");

            HttpResponse resp2 = client.execute(getMethod);
            String responseString2 = new BasicResponseHandler().handleResponse(resp2);
            System.out.println(responseString2);


        } catch (IOException e) {
            e.printStackTrace();
        }
}

Note: there is no authentication issue whatsoever, the page is returned normally besides that one detail.

score 0 · Answer 1 · answered Sep 25 '18 at 02:33

0

As far as I understand,

HttpPost postMethod = new HttpPost(baseUrl + "guardian/home.html");

home.html not containing any server site scripting, may you filled up your data by using javascript or something else by calling another api using Ajax call.

so from java,

HttpResponse resp2 = client.execute(getMethod);

it is taking the html and response the html,

answered Sep 25 '18 at 02:33

Mahamudul Hasan

2,745
2
17
26

Yes, the html page does contain JavaScript, but on other systems that don't run JavaScript (like requests.py or postman), the area is still populated with content. – dnsge Sep 25 '18 at 02:37
better if you share the home.html – Mahamudul Hasan Sep 25 '18 at 02:38

score 0 · Accepted Answer · answered Sep 26 '18 at 01:54

0

I ended up using Jsoup, following the exact same logic and got my desired result. Still no clue why HttpClient didn't work.

answered Sep 26 '18 at 01:54

dnsge

3
2
4

httpclient delivering different results

2 Answers2