nutch time schedule to visit a page again

Question

I have configured Nutch 2.3.1 with Hadoop/Hbase ecosystem. I have few hundred domains that I want to fetch. I have fetched many of them till now. I am curious that when Nutch will visit already fetched document again and refetch it if it is update. Is there any control parameter or something like that ?

Answer 1

Nutch has several ways to configure when a page is fetched again (see https://github.com/apache/nutch/blob/release-2.3.1/conf/nutch-default.xml).

db.fetch.interval.default (initial fetch value assigned when the page is fetched for the first time). Keep in mind that the default implementation (db.fetch.schedule.class, https://github.com/apache/nutch/blob/release-2.3.1/conf/nutch-default.xml#L396) always add the fetch interval to the last fetch time, so is not ideal. I would recommend switching to the adaptive fetch schedule algorithm, that will try to optimize the next fetch time depending on how often the page is updated (https://github.com/apache/nutch/blob/release-2.3.1/src/java/org/apache/nutch/crawl/AdaptiveFetchSchedule.java).

Keep in mind that you can also specify per URL fetch time (at inject time) using the nutch.fetchInterval metadata key in the seed file (https://github.com/apache/nutch/blob/release-2.3.1/src/java/org/apache/nutch/crawl/InjectorJob.java#L59).