How to get if an url is 404 or 301 in crawler4j

Question

Is it possible to get if an URL is 404 or 301 in crawler4j ?

@Override
    public void visit(Page page) {
        String url = page.getWebURL().getURL();
        System.out.println("URL: " + url);

        if (page.getParseData() instanceof HtmlParseData) {
            HtmlParseData htmlParseData = (HtmlParseData) page.getParseData();
            String text = htmlParseData.getText();
            String html = htmlParseData.getHtml();
            List<WebURL> links = htmlParseData.getOutgoingUrls();

            System.out.println("Text length: " + text.length());
            System.out.println("Html length: " + html.length());
            System.out.println("Number of outgoing links: " + links.size());
        }
    }

I use this in the crawler code .Can anyone tell me how ?

score 2 · Answer 1 · edited May 23 '17 at 12:27

As Crawler4j Version 3.3 (Feb 2012 released) - Crawler4j supportting for handling http status codes for fetched pages.

to visit StatusHandlerCrawlerExample click.

Also you can parse pages by using Jsoup (Java HTML Parser, with best of DOM, CSS, and jquery). And there is an example here - shows how to download a page from given URL and getting page status code. I think you should use Crawler4j for crawling and Jsoup for page fetching.

How to get if an url is 404 or 301 in crawler4j

1 Answers1