8

I want to submit my site to Google. How much time does it take to crawl a new post on the website?

Also, is there a way to feed this post to Google crawler as soon as a post is created?

Mike Hudson
  • 1,180
  • 5
  • 13
Usama Khalil
  • 436
  • 1
  • 4
  • 14
  • -1: I can't figure out what you are doing. Are you talking about google's crawler on SO, or are you talking about google's crawler on your site, or are you talking about telling google's crawler how often to re-index your site? –  Jan 26 '12 at 03:47
  • its a general question about google crawler. be it stackoverflow or any website that the bot crawls – Usama Khalil Jan 26 '12 at 04:04
  • This needs to be on the webmasters site. – the Tin Man Jan 27 '12 at 00:08

5 Answers5

7

Google has three modes of entering a website into its results - discover, crawl, index.

In order to 'discover' your site, it must be made aware of it's existence - normally through back-links. If you're site is brand new you can use the submit URL form - but this isn't really a trusted method. You're better off signing up for a Google Webmaster Tools account and submitting your site. An additional step is to submit an XML sitemap of your site. If you are publishing to your site in a blogging/posting way - you can always consider PubSubHubbub.

From there on, crawl frequency is normally based on site popularity (as measured by ye olde PageRank). Depth of crawl (crawl-budget) is also determined by PR.

Mike Hudson
  • 1,180
  • 5
  • 13
  • marked as answer as google itself suggest the XML site map submission – Usama Khalil Feb 10 '12 at 03:02
  • how much is limit to upload site map? is that based on the upload file size or the number of entries in a file? – Usama Khalil Mar 19 '12 at 08:11
  • +1 For the point re: crawl times and PR - You'll note that SO Questions appear almost instantly, whereas less popular sites can take days/weeks for new content to appear. – Basic Aug 06 '12 at 12:18
0

For fast crawl you should submit your xml sitemap in google web master and manually crawled and index your web pages url through google webmaster fetch.
I also used google crawled and index method and after that this practices give me best result.

bummi
  • 27,123
  • 14
  • 62
  • 101
Kshitij Bhardwaj
  • 1
0

This is a great resource that really breaks down all the factors that affect a crawl budget and how to optimize your website to increase it. Cleaning up your broken links and removing outdated content, for example, can work wonders. https://prerender.io/crawl-budget-seo/ 

Zoltan Bettenbuk
  • 1
  • 1
0

There are a couple ways to help "feed" the Google Crawler a URL.

The first way is to go here and submit a URL ---> www.google.com/webmasters/tools/submit-url/

The second way is to go to your Google Webmasters Tools and clicking "Fetch as GoogleBot"

And then inputting the URL you want to add:

https://i.stack.imgur.com/Q3Iva.png

The URL will then appear similar to this:

http:\\example.site Web Success URL submitted to index 1/22/12 2:51 AM

As for how long it takes for a question on here to appear on google, there are many factors that are put in to this.

If the owners of the site use Google Webmasters Tools, the following setting is available:

https://i.stack.imgur.com/RqvOi.png

anonymous
  • 1,111
  • 1
  • 10
  • 18
  • 1
    Can't agree with `Revisit-After` - see Google's own comments here - http://code.google.com/webstats/2005-12/metadata.html, and more here: http://googlewebmastercentral.blogspot.com/2007/12/answering-more-popular-picks-meta-tags.html – Mike Hudson Jan 26 '12 at 22:22
  • Mike, thank for point out resources for Revisit-after. Wow, I didn't know this was not being used. I also liked your second link so giving you a thumbs up. :-) – smallworld Jan 28 '12 at 16:50
-1

I acknowledged error in my response by adding a comment to original question a long time ago. Now, I am updating this post in interest of keeping future readers from being misguided as I was. Please see notes from other users below - they are correct. Google does not make use of the revisit-after meta tag. I am still keeping the original response text here to make sure that anyone else looking for similar answer will find it here along with this note confirming that this meta tag IS NOT VALID! Hope this helps someone.

You may use HTML meta tag as follows: 

<meta name="revisit-after" content="1 day">

Adjust time period as necessary. There is no guarantee that robots will return in given time frame but this is how you are telling robots about how often a given page is likely to change.

  • The Revisit Meta Tag is used to tell search engines when to come back next.
smallworld
  • 910
  • 7
  • 15
  • Ok, I think you are looking for Google Custom Search integration into your site and controlling frequency at Google will index your site. I think better description (use at least the product/service name as provider uses) would have helped. You will find answers you are looking for here .... Updating pages for on-demand indexing - [link](http://support.google.com/customsearch/bin/answer.py?hl=en&answer=116420&ctx=cb&src=cb&cbid=-5yovkk5jsvky&cbrank=1) – smallworld Jan 26 '12 at 04:04
  • Disagree with this, see other comment. – Mike Hudson Jan 26 '12 at 22:25
  • Google does not utilize this tag, see http://googlewebmastercentral.blogspot.com/2007/12/answering-more-popular-picks-meta-tags.html – Jamol Aug 03 '12 at 13:38