5

I try to get a proxy list from this url:

Free proxy list

This would be cool, but port number is dynamic JavaScript content. How can I get JavaScript-generated content from this page? I have jsoup and djNativeSwing but I want do this in background thread. 

JWebBrowser webBrowser = new JWebBrowser();
webBrowser.navigate("http://spys.ru/en/free-proxy-list/");
System.out.println(webBrowser.getHTMLContent());

this code returns a Null result. Help please.

Lee Taylor
  • 7,761
  • 16
  • 33
  • 49
Trump
  • 77
  • 1
  • 7
  • check out `htmlUnit` I think it is the one thing that you need. Thanks. – Dhruvenkumar Shah Aug 23 '12 at 17:57
  • i think this is not for me. My code must be simple and crossplatform. As i know HTMLUnit - must have SYSTEM path - it's trouble – Trump Aug 23 '12 at 18:13
  • No you do not need system path. I created web crawler using HTMLUnit which works on http protocol. so yes it does work properly. Let me know if you want more help. I can write a formal answer if you want. – Dhruvenkumar Shah Aug 23 '12 at 18:23
  • Ohhh! Give me example! I try to use this! – Trump Aug 23 '12 at 18:27
  • I'm try to use this code: final WebClient webClient = new WebClient(); final HtmlPage page = webClient.getPage("http://spys.ru/free-proxy-list1/RU/"); System.out.println(page.toString()); – Trump Aug 23 '12 at 18:31
  • And get this error: Exception in thread "Thread-2" java.lang.NoClassDefFoundError: org/apache/http/client/CredentialsProvider at sinonimizer.PGinside.getProxyInfo(PGinside.java:60) at sinonimizer.PGinside.run(PGinside.java:50) Caused by: java.lang.ClassNotFoundException: org.apache.http.client.CredentialsProvider at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadCla – Trump Aug 23 '12 at 18:33
  • What is this? All libs added in project... – Trump Aug 23 '12 at 18:35
  • htmlunit lib, 5 commons, cssparser lib... anything else? – Trump Aug 23 '12 at 18:48

1 Answers1

2

The webbrowser hasn't finnished loading when you call the getHtmlContent() method. Use something like this instead: 

JWebBrowser webBrowser = new JWebBrowser();
webBrowser.navigate("http://spys.ru/en/free-proxy-list/");
webBrowser.addWebBrowserListener(new WebBrowserListener(){
   public void loadingProgressChanged(WebBrowserEvent e){
       if(e.getWebBrowser().getLoadingProgress()==100)
            System.out.println(webBrowser.getHTMLContent());
   }
}
/* Note: I wrote this in the comment field without any testing,
   you probably have to make the webBrowser final. */

JavaDocs is your friend!

Henrik Karlsson
  • 5,559
  • 4
  • 25
  • 42