HtmlUnit Page response (Downloading File)

Question

So im trying to use HtmlUnit to go to a URL but once you visit that url it downloads a json file regarding the data you want. Not sure how to word this but basically in HtmlUnit how can I get the result from a downloaded file.

I suck at explaining here look

trying to check user availability by this

private static final String URL = "https://twitter.com/users/username_available?username=";

...

HtmlPage page = webClient.getPage(URL + users[finalUsersIndex]);

so that basically creates a new page for each username thing is the URL + username returns a json file of user availability. I know how to read the json file but the problem is this

java.lang.ClassCastException: com.gargoylesoftware.htmlunit.UnexpectedPage
 cannot be cast to com.gargoylesoftware.htmlunit.html.HtmlPage

I get that on this line

HtmlPage page = webClient.getPage(URL + users[finalUsersIndex]);

I suppose I need to create a new page for the response but how would I do that since it automatically downloads file instead of per se, clicking a button which downloads the file. (Correct me if im wrong)

Sorry 4AM

Answer 1

As its name indicates, an HtmlPage is a page containing HTML. JSON is not HTML.

As the documentation indicates;

The DefaultPageCreator will create a Page depending on the content type of the HTTP response, basically HtmlPage for HTML content, XmlPage for XML content, TextPage for other text content and UnexpectedPage for anything else.

(emphasis mine).

So, as the exception you're getting indicates, the behavior you observed is the documented behavior: you're getting a page that is neither HTML, nor XML, nor text, so you get an UnexpectedPage.

Your code should thus be:

UnexpectedPage page = webClient.getPage(URL + users[finalUsersIndex]);