Count the number of pages in a site

Question

I'd like to know how many public pages there are in a site, say for example, smashingmagzine.com. Is there are way to count the number of pages?

score 3 · Answer 1 · answered Feb 09 '11 at 14:16

You can query Google's index using the site operator. e.g:

site:domain-to-query.com

This will return a list of the pages from the site that are currently indexed by Google. Other search engines provide similar functionality but I don't know the syntax off hand.

Of course not all pages may be indexed, and the index may contain pages which no longer exist.

score 2 · Answer 2 · answered Feb 09 '11 at 14:13

2

You need to basically crawl the site. Your process would be something like:

Start at root domain / homepage
Look for all links that point within the same domain
For each of those links, repeat the steps

Your loop terminates when there are no more links to crawl that are pointing in the same domain. Remember to stay in the site otherwise you'll start crawling external sites.

You can also try parsing the sitemap if they provide one.

One tool that might prove useful if using Java is JSpider or Sphider in PHP.

answered Feb 09 '11 at 14:13

NG.

22,560
5
55
61

How would I check whether I'm still on the same domain? RegEx? – Gaurav Sharma Feb 09 '11 at 15:05
You can look at the hosts of the URL being linked to. A URL parser can easily get you this info in PHP. – NG. Feb 09 '11 at 16:10

score 0 · Answer 3 · answered Feb 09 '11 at 14:14

You'll need to recursively scan the markup of each page, starting with your top level page, looking for any kind of links to other pages, and recursively crawl through them. You'll also need to keep track of what has been scanned as to not get caught in an infinate loop.

Count the number of pages in a site

3 Answers3