Open a connection with Jsoup, get status code and parse document

Question

I'm creating a class using jsoup that will do the following:

1. The constructor opens a connection to a url.
2. I have a method that will check the status of the page. i.e. 200, 404 etc.
3. I have a method to parse the page and return a list of urls.#

Below is a rough working of what I am trying to do, not its very rough as I've been trying a lot of different things

public class ParsePage {
private String path;
Connection.Response response = null;

private ParsePage(String langLocale){
    try {
        response = Jsoup.connect(path)
                .userAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21")
                .timeout(10000)
                .execute();
    } catch (IOException e) {
        System.out.println("io - "+e);
    }
}

public int getSitemapStatus(){
    int statusCode = response.statusCode();
    return statusCode;
}

public ArrayList<String> getUrls(){
    ArrayList<String> urls = new ArrayList<String>();

 }
}

As you can see I can get the page status, but using the already open connection from the constructor I don't know how to get the document to parse, I tried using:

Document doc = connection.get();

But that's a no go. Any suggestions? Or better ways to go about this?

Answer 1

As stated in the JSoup Documentation for the Connection.Response type, there is a parse() method that parse the response's body as a Document and returns it. When you have that, you can do whatever you want with it.

For example, see the implementation of getUrls()

public class ParsePage {
   private String path;
   Connection.Response response = null;

   private ParsePage(String langLocale){
      try {
         response = Jsoup.connect(path)
            .userAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21")
            .timeout(10000)
            .execute();
      } catch (IOException e) {
         System.out.println("io - "+e);
      }
   }

   public int getSitemapStatus() {
      int statusCode = response.statusCode();
      return statusCode;
   }

   public ArrayList<String> getUrls() {
      ArrayList<String> urls = new ArrayList<String>();
      Document doc = response.parse();
      // do whatever you want, for example retrieving the <url> from the sitemap
      for (Element url : doc.select("url")) {
         urls.add(url.select("loc").text());
      }
      return urls;
   }
}

Answer 2

If you don't need to login, use:

Document doc = Jsoup.connect("url").get();

If you DO need to login I'd advise using:

Response res = Jsoup.connect("url")
    .data("loginField", "yourUser", "passwordField", "yourPassword")
    .method(Method.POST)
    .execute();
Document doc = res.parse();

//If you need to keep logged in to the page, use
Map<String, String> cookies = res.cookies;

//And by every consequent connection, you'll need to use
Document pageWhenAlreadyLoggedIn = Jsoup.connect("url").cookies(cookies).get();

In your usage to get urls I'd probably try

Elements elems = doc.select(a[href]);
for (Element elem : elems) {
    String link = elem.attr("href");
}

That's about it.. Keep up the good work

Answer 3

You should be able to call parse() on your response object.

Document doc = response.parse();

Answer 4

Seems your situation like you want to make connection with jsoup then check the status code and then according to the status code you will parse or whatever you want to do.

For this first you have to check the status code of the URL instead creating connection.

  Response response = Jsoup.connect("Your Url ").followRedirects(false).execute();
        System.out.println(response.statusCode() + " : " + response.url());

response.statusCode() will return you the status code

After that you can create your connection

 if (200 == response.statusCode()) {
        doc = Jsoup.connect(" Your URL").get();
        Elements elements = doc.select("href");
        /* what ever you want to do*/
      }

Your class will look like this

package com.demo.soup.core;

import java.io.IOException;

import org.jsoup.Connection.Response;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

/**
 * The Class DemoConnectionWithJsoup.
 *
 * @author Ankit Sood Apr 21, 2017
 */
public class DemoConnectionWithJsoup {

    /**
     * The main method.
     *
     * @param args
     *            the arguments
     */
    public static void main(String[] args) {
    Response response;
    try {
        response = Jsoup.connect("Your URL ").followRedirects(false).execute();

        /* response.statusCode() will return you the status code */
        if (200 == response.statusCode()) {
        Document doc = Jsoup.connect("Your URL").get();

        /* what ever you want to do */
        }
    } catch (IOException e) {
        e.printStackTrace();
    }

    }

}