2

I want to know if there are any outsource solutions for a crawler that will parse only the links and pages form a given website, and will output:

1.The link tree 2.The pages (where necessary)

thanks!

dana
  • 5,168
  • 20
  • 75
  • 116

1 Answers1

0

You dont need any particular framework to achieve this task. What languages do you know? If you know Java you can use HttpClient or HttpUnit libs to help you with crawling tasks.

If you are python user, there is great framework called Scrapy (http://scrapy.org/). You should check it out.

Fabio
  • 2,074
  • 2
  • 24
  • 38
svlada
  • 3,218
  • 2
  • 25
  • 36