FailingHttpStatusCodeException: 401 Unauthorized acess using HtmlUnit

Question

I'm trying to connect to a page using HtmlUnit using this code:

WebClient webClient = new WebClient();
webClient.getOptions().setThrowExceptionOnScriptError(false);
HtmlPage loginPage = webClient.getPage(WEBSPHERE_URL);

But I'm always getting this error:

18:06:00,390 ERROR [stderr] (Refresh Thread: Equinox Container: 90609616-a614-0017-138e-e6e4aa4a0871) com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException: 401 Unauthorized for https://portal-intranet.ti.sabesp.com.br/wps/myportal
18:06:00,390 ERROR [stderr] (Refresh Thread: Equinox Container: 90609616-a614-0017-138e-e6e4aa4a0871)   at com.gargoylesoftware.htmlunit.WebClient.throwFailingHttpStatusCodeExceptionIfNecessary(WebClient.java:576)
18:06:00,390 ERROR [stderr] (Refresh Thread: Equinox Container: 90609616-a614-0017-138e-e6e4aa4a0871)   at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:401)

When I try access the URL using a browser, it works perfectly. How should I do to get this problem fixed?

UPDATE

Now I'm getting this error:

21:43:35,746 ERROR [osgi.logging.setup_environment] (Thread-257) [br.com.sabesp.setupenvironment.portlet.SetupEnvironmentAction(2427)] The activate method has thrown an exception : java.lang.LinkageError: loader constraint violation: when resolving method "com.gargoylesoftware.htmlunit.WebClient.setCredentialsProvider(Lorg/apache/http/client/CredentialsProvider;)V" the class loader (instance of org/eclipse/osgi/internal/loader/EquinoxClassLoader) of the current class, br/com/sabesp/setupenvironment/importer/util/WebContentUtil, and the class loader (instance of org/eclipse/osgi/internal/loader/EquinoxClassLoader) for the method's defining class, com/gargoylesoftware/htmlunit/WebClient, have different Class objects for the type org/apache/http/client/CredentialsProvider used in the signature

When I try do this:

DefaultCredentialsProvider userCredentials = new DefaultCredentialsProvider();
userCredentials.addCredentials("username", "password");
webClient.setCredentialsProvider(userCredentials);

I'm working with Liferay DXP witch is based on OSGi framework and I'm using also Gradle.

Answer 1

You seem to run against an intranet website, and this can detect your username from the browser I guess.

With HtmlUnit, you can use NTLM authentication:

DefaultCredentialsProvider credentialsProvider =
                (DefaultCredentialsProvider) webClient.getCredentialsProvider();
credentialsProvider.addCredentials("username", "password");
//OR
credentialsProvider.addNTLMCredentials("username", "password", null, -1, "localhost", "domain");

You can also enable traffic logging to see what type of authentication is needed:

LogManager.getLogger("org.apache.http.wire").setLevel(Level.ALL);

Update :

You seem to have conflicting httpclient jars, please remove one of them.

The following code would tell which one is being used:

System.out.println(CredentialsProvider.class.getProtectionDomain()
    .getCodeSource().getLocation());