Nutch - Visit few pages again and again to find new links

Asked Aug 26 '20 at 01:57

Active Aug 26 '20 at 01:57

Viewed 66 times

I have setup Nutch 1.17 to crawl few thousand domains with inlinks crawl only. One of my main requirement is I should have to visit home pages again and again (lets say after 2 hour) and if there is any new page, then only that should be crawled.

What should be the best possible way ? I am thinking to crawl run injector job again and again to crawl home pages. Is it the right way ? Meanwhile how should I ensure that inlinks are also going to fetch with time.

asked Aug 26 '20 at 01:57

Hafiz Muhammad Shafiq

8,168
12
63
121

Are you able to find the solution – Ravi Kiran Nov 05 '20 at 09:38
Yes, I found a solution. In seed, you should specify the revisit time e.g., 4 hours. Then, those URLs will be selected again after 4 hours. – Hafiz Muhammad Shafiq Nov 05 '20 at 10:35
may i know the parameter in nutch – Ravi Kiran Nov 06 '20 at 11:38
May I know are you using the nutch webui @Shafiq – Ravi Kiran Nov 06 '20 at 15:57
May I know how nutch knows new page is added then crawl it.Can you help me. – Ravi Kiran Nov 06 '20 at 23:53
In seed file, lets say you want to recrawl seed URL, after 60 minute. Then this will be the seed file format. http://humkinar.pk/ nutch.fetchInterval.fixed=60 userType=open_source – Hafiz Muhammad Shafiq Nov 07 '20 at 02:43
During fetching, new links are found that are parsed in parser job. Then updatedb job simply create a new record (row) in the db. From this DB, new URLs are selected based on their meta info. – Hafiz Muhammad Shafiq Nov 07 '20 at 02:45
Please spend some time on Nutch documentation/manual. – Hafiz Muhammad Shafiq Nov 07 '20 at 02:46
may I know what is modified time in nutch i see some columns as modified time. – Ravi Kiran Nov 07 '20 at 12:34
Thanks for info.may i know what is difference between db.fetch.interval.default andnutch.fetchInterval.fixed. I crawled some urls it marked as retry fetch interval is 30 days and db_unfecthed urls as retryintervaldays is 30 days but when crawl next round db_gone as retry days is 45 days on what basis this incremented the 15 days. – Ravi Kiran Nov 07 '20 at 12:59
bin/nutch inject $CRAWL_PATH/crawldb http://wisconsinrealtyexecutives.com nutch.score=100 nutch.fetchInterval.fixed=10 userType=open_source -update It is not working it is showing like this 20/11/09 11:33:53 INFO crawl.Injector: Injector: Found invalid argument "nutch.score=100" – Ravi Kiran Nov 09 '20 at 11:34
May i know what is wrong in the command. – Ravi Kiran Nov 09 '20 at 11:36
These arguments should be tab separated. Please have a look on Nutch wiki for this – Hafiz Muhammad Shafiq Nov 09 '20 at 11:40
Thanks I want to remove the urls which are not crawlable and not working from crawldb. – Ravi Kiran Nov 10 '20 at 05:29
Brother I have to check it also. No idea at this time – Hafiz Muhammad Shafiq Nov 10 '20 at 07:10
Will nutch shows the next fetch time for urls because i can't see the next fetch time. – Ravi Kiran Nov 10 '20 at 11:28

Nutch - Visit few pages again and again to find new links

0 Answers0