0

I'm doing a webcreawler and now I'm trying to follow the robots.txt and the metatags NOINDEX and NOFOLLOW.

NOFOLLOW says me to not follow the links in the page, but and the NOINDEX?
I understand that it's to not index the page on the searches, but I won't do any search, only store the page in the database.

I will disrespect the NOINDEX tag if I store in the database or this only applies to searches?

Renato Dinhani
  • 35,057
  • 55
  • 139
  • 199

1 Answers1

0

No particular reason not to store it. As long as you're not indexing it and returning it in a search result. You're not disrespecting the meaning of NOINDEX unless you return it as a search result.

Jim Mischel
  • 131,090
  • 20
  • 188
  • 351