0

XML sitemap generator plugin simply put the following string in robot.txt file, if we see so many wp blogs they have lots of tags included in it. also my xml file looks like "sitemap.xml.gz" this,

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

How to edit and put what are the possible parameters we need to put in robot.txt file?. is the xml file "sitemap.xml.gz" name valid, what is the difference between .xml and gz file?

unor
  • 92,415
  • 26
  • 211
  • 360
Naruto
  • 9,476
  • 37
  • 118
  • 201
  • The `.xml.gz` file is a gzipped equivalent of the `.xml` file with the same name, and, depending on your Apache configuration, will be preferentially served, with a `Content-Encoding: gzip` header, when the `.xml` file is requested. This saves bandwidth for both the server and the client. As for what you need to put in your `/robots.txt` file, my answer would be 'nothing'; just make sure its permissions are set writable for the web server user account, so that Wordpress can modify it according to the settings you make in the Wordpress control panel. – Aaron Miller Jun 10 '13 at 16:44
  • (Your sitemap plugin, which isn't linked from your question but should be, may require manual editing of robots.txt; if so, follow whatever instructions it provides. In general, though, Wordpress will do quite a respectable job of managing robots.txt content for you.) – Aaron Miller Jun 10 '13 at 16:45
  • Thanks Aaron, but also i have one more query. I am seeing my post takes nearly 2 days of time to appear in search engine. where could be the problem? theme, plugin or wat? any idea? – Naruto Jun 11 '13 at 10:38
  • That's probably just down to the time it takes Google to index your site; I've seen times considerably greater than two days. Not sure what advice to give there, sorry. – Aaron Miller Jun 11 '13 at 14:24
  • Also one more thing i wanted to ask, can we increase the crawl speed in webmaster tools?. Also the priority of posts in xml sitemap states what?, can we change that to improve the crawl speed?. if so please advice me – Naruto Jun 12 '13 at 04:30
  • I'd love to be able to help you, but have never worked with webmaster tools directly. – Aaron Miller Jun 13 '13 at 05:24
  • Oh, thank you verymuch. If i need any help, ill drop a comment for you please help me. Thanks lot – Naruto Jun 13 '13 at 08:58

0 Answers0