Several website like Quora, Stackechange, and including Stackoverflow (https://stackoverflow.com/sitemap.xml) only access through the search engine crawlers (Google, Yahoo, Bing, etc).
- How can i do same for my website robots.txt and sitemap.xml
- What are the user-agents these crawlers use and where i can find a list
- Google and Bing crawlers do not use any static IP's, they are dynamic and lot of IP's. How this big site like Stackoverflow manage whitelisting IP's of crawlers.
- How big site content indexed instantly on Google. like my this question will get indexed instantly after publishing it. where my website usually take 2-7 days for indexing.
