What is the best Open Source Web Crawler Tool written in Java?

Question

What is the best Open Source Web Crawler Tool, written in Java.

score 10 · Accepted Answer · edited Jul 15 '16 at 20:21

10

Try crawler4j. You just need to implement a simple interface which controls which URLs to visit and what to do with each crawled page.

edited Jul 15 '16 at 20:21

Scott Wardlaw

answered Dec 12 '11 at 12:20

Andy

I have problems crawling HTTPS websites using this crawler ("site failed to respond" while it opens fine in browser etc.) – ed22 Sep 29 '17 at 08:29

score 5 · Answer 2 · answered Dec 12 '11 at 12:20

5

in java I think it boils down to Nutch vs Heritrix. You should specify what your needs are to get a better answer.

answered Dec 12 '11 at 12:20

riffraff

2 Answers2