2

I need some help. I'm not sure about the order on request for mod_rewrite and robots.txt.

Some urls belong to a rewrite rule:

/index.php?id=123 to /home

Other urls don't have a rewrite:

/index.php?id=444

I made this entry to my robots.txt:

User-agent: *
Disallow: /index.php?id

Will the site with /home be indexed by search engines?

Christoph
  • 1,113
  • 5
  • 17
  • 35

1 Answers1

4

The robots.txt file is interpreted by the client (spider), and they don't know what rewrites you have in your system. Thus, spiders would not fetch URLs from your site if they look like the pattern in robots.txt but would if they found the same content through /home.

Jeremiah Willcock
  • 30,161
  • 7
  • 76
  • 78