How to get the "Title" from a webpage using HttpClient

Question

I'm trying to get the "Title" from a webpage using Apache HttpClient 4.

Edit: My first approach was trying to get it from the header (using HttpHead). If that is not possible, how can I get it from the body of the response, as @Todd says?

Edit 2:

<head>
[...]
<title>This is what I need to get!</title>
[...]
</head>

Are you sure your target server is sending back a "title" header? — tddmonkey, Jan 23 '15 at 14:34
The title isn't in the response headers, it's part of the HTML returned in the body of the response, if it's there at all. — Todd, Jan 23 '15 at 14:38
@MrWiggles Yes. I'm pretty sure. I'm testing with http://www.ibm.com and it has a tag inside the section. — Mario S, Jan 23 '15 at 15:00
Check http://jsoup.org/ for a decent Java based HTML parser/scraper — Alex, Jan 23 '15 at 15:08
@Alex You're right too. I'm going to put the solution below. Thank you too. — Mario S, Jan 23 '15 at 15:15
Please update the title of your question to reflect your needs so it's not misleading for anyone coming to this afterwards — tddmonkey, Jan 23 '15 at 15:16

Mario S · Accepted Answer · 2015-01-23T15:49:20.600

Thank you everyone for your comments. The solution was pretty simple once jsoup was used.

Document doc = Jsoup.connect("http://example.com/").get();
String title = doc.title();

Considering that I really need to connect using HttpClient, this is what I have:

org.jsoup.nodes.Document doc = null;
String title = "";

System.out.println("Getting content... ");

CloseableHttpClient httpclient = HttpClients.createDefault();
HttpHost target = new HttpHost(host);
HttpGet httpget = new HttpGet(path);
CloseableHttpResponse response = httpclient.execute(target, httpget);

System.out.println("Parsing content... ");

try {
    String line = null;
    StringBuffer tmp = new StringBuffer();
    BufferedReader in = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
    while ((line = in.readLine()) != null) {                    
        String decoded = new String(line.getBytes(), "UTF-8");
        tmp.append(" ").append(decoded);
    }

    doc = Jsoup.parse(String.valueOf(tmp)); 

    title = doc.title();
    System.out.println("Title=" + title); //<== ^_^

    //[...]

} finally {
    response.close();
}

System.out.println("Done.");

In case you see http 403 with above code, add .userAgent("Mozilla") like Document doc = Jsoup.connect("http://example.com/").userAgent("Mozilla").get(); String title = doc.title(); — Vaibs, Mar 19 '17 at 13:33

score 1 · Answer 2 · answered Apr 30 '19 at 03:38

By using this code fragment, you still can retrieve the <title> of the web page by proving its URL.

InputStream response = null;
    try {
        String url = "http://example.com/";
        response = new URL(url).openStream();


        Scanner scanner = new Scanner(response);
        String responseBody = scanner.useDelimiter("\\A").next();
        System.out.println(responseBody.substring(responseBody.indexOf("<title>") + 7, responseBody.indexOf("</title>")));

    } catch (IOException ex) {
        ex.printStackTrace();
    } finally {
        try {
            response.close();
        } catch (IOException ex) {
            ex.printStackTrace();
        }
    }

How to get the "Title" from a webpage using HttpClient

2 Answers2