error 403 in htmlunit scraping although it’s manually opening

Question

I’m trying to use html_unit 1.14 for scraping on fake proxy. It gives me (error 403), although it’s opened manually on the browser.

What could be the reason and how to detect it?

Answer 1

Step 1 (optional) : upgrade to v2.18 or newer

<dependency>
    <groupId>net.sourceforge.htmlunit</groupId>
    <artifactId>htmlunit</artifactId>
    <version>2.18</version>
</dependency>

Step 2 (mandatory) :

WebClient webClient = new WebClient(yourbrowser);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);