0

I need to analyze the search engine crawling going on in my site. Is there a good tool for this? I've tried AWStats and Sawmill. But both of those give me very limited insight into the crawling. I need to know information like how many unique/distinct webpages in a section of my site was crawled by a specific crawler within a time period.

Google analytics doesn't track crawling at all due to its javascript tracking mechanism.

apptree
  • 345
  • 1
  • 3
  • 10

1 Answers1

0

Clarity is a front end interface similar to Splunk.

Logstash should be able to help you with what you need.

mt3
  • 308
  • 1
  • 3
  • 12