Error using HtmlUnit 2.18 for crawl webpages

Question

I have the following code:

WebClient webClient = new WebClient(BrowserVersion.getDefault());
HtmlPage page;
List<HtmlAnchor> anchor=new ArrayList<HtmlAnchor>();

try {
    System.out.println("Querying");
    page = webClient.getPage("https://www.amazon.com/gp/goldbox");
    anchor = page.getAnchors();
    for(HtmlAnchor s:anchor)
    {
      System.out.println(s.getAttribute("href"));
    }
    System.out.println("Success");
}

Querying

Exception in thread "main" java.lang.NoSuchFieldError: INSTANCE
    at org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<init>(DefaultHttpRequestWriterFactory.java:52)
    at org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<init>(DefaultHttpRequestWriterFactory.java:56)
    at org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<clinit>(DefaultHttpRequestWriterFactory.java:46)
    at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<init>(ManagedHttpClientConnectionFactory.java:82)
    at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<init>(ManagedHttpClientConnectionFactory.java:95)
    at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<init>(ManagedHttpClientConnectionFactory.java:104)
    at org.apache.http.impl.conn.ManagedHttpClientConnectionFactory.<clinit>(ManagedHttpClientConnectionFactory.java:62)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.<init>(PoolingHttpClientConnectionManager.java:572)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:174)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:158)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:149)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.<init>(PoolingHttpClientConnectionManager.java:125)
    at com.gargoylesoftware.htmlunit.HttpWebConnection.createConnectionManager(HttpWebConnection.java:972)
    at com.gargoylesoftware.htmlunit.HttpWebConnection.getResponse(HttpWebConnection.java:161)
    at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1321)
    at com.gargoylesoftware.htmlunit.WebClient.loadWebResponse(WebClient.java:1238)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:346)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:415)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:400)
    at crawler.HtmlUnitCrawl.main(HtmlUnitCrawl.java:29)

What might be the mistake?

score 0 · Answer 1 · answered Oct 31 '15 at 11:21

0

You have conflicting CLASSPATH, as your code works fine with a clean installation.

Please remove all HttpComponents .jars, and use what is provided by HtmlUnit.

Also, you can see which version is used by:

    Class<?> klass = DefaultHttpRequestWriterFactory.class;
    String location = klass.getProtectionDomain().getCodeSource().getLocation().toString();
    System.out.println(location);

Which should give the location of httpcore-4.4.1.jar in your case.

answered Oct 31 '15 at 11:21

Ahmed Ashour

5,179
10
35
56

I am getting error at this line: Class> klass = DefaultHttpRequestWriterFactory.class;. The error is "DefaultHttpRequestWriterFactory cannot be resolved as a Type" – TheGaME Oct 31 '15 at 14:21
`org.apache.http.impl.io.DefaultHttpRequestWriterFactory` should be in your CLASSPATH, isn't it? – Ahmed Ashour Oct 31 '15 at 18:05

score 0 · Answer 2 · answered Aug 29 '18 at 20:29

I verified that HtmlUnit use a version already used in my project. So I put the HtmlUnit version compatible with my project version and all worked fine.

Httpclient-4.2.1 was conflicting with HtmlUnit-2.21 (that uses httpclient-4.5.2.jar). So I change to HtmlUnit 2.10 (that uses Httpclient-4.2.1) and it worked fine.

Check what libs are conflicting in your project

Error using HtmlUnit 2.18 for crawl webpages

2 Answers2