1

I want crawler4j to visit pages in such a manner that they belong to domain in seed only. There multiple domains in seed. How can I do it?

Suppose I am adding seed URLs:

  • www.google.com
  • www.yahoo.com
  • www.wikipedia.com

Now I am starting the crawling but I want my crawler to visit pages (just like shouldVisit()) only in above three domains. Obviously there external links, but I want my crawler to restrict to these domains only. Sub-domain, sub-folders are okay, but not outside these domains.

halfer
  • 19,824
  • 17
  • 99
  • 186
akshayb
  • 1,219
  • 2
  • 18
  • 44
  • if you implement restricted url, if done can you post the answer it will help lot of people – Selva May 18 '15 at 12:19

2 Answers2

1

Posted on behalf of the OP:

Got the solution here : http://code.google.com/p/crawler4j/issues/detail?id=94#c1

halfer
  • 19,824
  • 17
  • 99
  • 186
  • issue on Github are empty & no description providen – alexandre-rousseau Jan 09 '18 at 06:54
  • @RousseauAlexandre: I just reposted this for the question author. You could comment under the question to see if they have more information for you. However they have not signed in since 2015, so you may have to ask a new question. – halfer Jan 09 '18 at 15:25
0

If you are trying to restrict the crawler to only urls with the same domains as the seed urls, then:

  1. Extract the domain names from the seed URLs.

  2. Write your crawler class (that extends WebCrawler) with a shouldVisit method to filter out any URLs whose domains are not in the set.

  3. Configure the controller, add the seeds and start it in the normal way ... as per the example here.

Stephen C
  • 698,415
  • 94
  • 811
  • 1,216
  • 1
    thanks. How should I pass domain names of seed urls from controller to crawler class function shouldVisit ? If number of seed urls are too large (in thousands), it will be memory intensive I think. Is it possible to crawl seeds one by one ?( I am following the basic crawler example). – akshayb Nov 09 '13 at 11:39
  • I have asked another query that discusses an alternative approach here : http://stackoverflow.com/questions/19875771/calling-controller-start-in-loop-in-crawler4j – akshayb Nov 09 '13 at 12:11