Questions tagged [google-search]

Google is the most popular search engine in the world. The Google Web Search API has been deprecated in favor of the new Custom Search.

A Google search may not return answers that might be expected for reasons that include those mentioned in answers and comments to What can you NOT find on Google?:

Google does not even attempt

• To search for a keywords that are special characters:

"Generally, punctuation is ignored, including @#$%^&*()=+[]\ and other special characters" -Franck Dernoncourt.

The search term double unary works but not --. See also Google displays my website as a spelling error.

Sites with too much content, with content of little value or that are impractical to index

May include:

• Sites that don't have a crawlable site map and require google to provide search terms to access the results available on the site might not be fully indexed. -Josephine Bonaparte
• Smaller blogs that aren't regularly updated are often dumped from search results. Plus anything that they think is a splog (“a blog which the author uses to promote affiliated websites” -Wikipedia). -David
• “Most of the Twitter content is not indexed by Google, even if it’s public.
  It used to be available to Google, but that’s no longer the case since their agreement expired.” -Alex
• “Google does not index Tumblr all that well.
  Blog posts on Tumblr are easier to find using Tumblr search.” -David
• “everything on Google Sites isn't (or is hardly) indexed.
  If you start a Google site, get your own domain.” -David

Copyright and other protected material

May include:

• What the government thinks is not good for you. –Hellagot
  The example give was of Germany “does not show thousands of sites … and the list increases by the thousands every year”.
• What may infringe intellectual property rights. –einpoklum
  DMCA (Digital Millennium Copyright Act) was mentioned.
• Census images.
  “Since the content are images that are often manually index, they usually found on paid-for sites like ancestry.com.” –amh

To see which URLs Google has been blocked from crawling, visit the Blocked URLs page of the Crawl section of Webmaster Tools.

Opt outs

• Content explicitly disallowed by a domain's robots.txt file is excluded from the Google index. -amh

Technical complications

• Websites that are not linked from other websites that Google already knows (perhaps from when domain was under different ownership – Tim Post). That is, there are probably a lot of websites that do not get linked from visible pages, those websites are never going to be found by the Google spider unless they're manually submitted to Google via the Webmaster Tools. –amh
• Websites that are behind web forms that you need to fill out. –amh
• The Deep Web “Most of the Web's information is buried far down on dynamically generated sites, and standard search engines do not find it. Traditional search engines cannot "see" or retrieve content in the deep Web—those pages do not exist until they are created dynamically as the result of a specific search. As of 2001, the deep Web was several orders of magnitude larger than the surface Web.” -Wikipedia
• May include 408 Billion web pages saved over time according to Wayback Machine. –pnuts