5

I want to publish an Angular app for testing purposes, but I want to make sure that the site does not get crawled or indexed by bots.

I assume (might be way off!) I would add my <meta> tags simply on my index.html page, and for good measure add a robots.txt file in my root?

These are my meta tags:

<meta name="robots" content="noindex,nofollow">
<meta name="googlebot" content="noindex" />

This is the content of my robots.txt file:

User-agent: *
Disallow: /

Thank you in advance!

onmyway
  • 1,435
  • 3
  • 29
  • 53
  • 2
    The solution which you have mentioned should work. meanwhile you can checkout this. https://stackoverflow.com/questions/9102914/how-to-stop-search-engines-from-crawling-the-whole-website. and you set some rules in hosting site to block IP's to access your site. – Allabakash Oct 28 '19 at 09:18

1 Answers1

3

Using the robots.txt file you specified will be enough to prevent your site from being indexed by the bots that follow the robots exclusion standard. With this robots.txt you don't need to specify the meta headers, because the bot read the robots.txt first and won't parse HTML of the website to read the meta tags.

The meta tags are used when your robots.txt file would normally allow to index that page, but you want to exclude it on the page-level, which allows more granular selection.

Note that some uncommon crawlers may not respect the exclusion standard. If you really want to restrict access to your test site, you should consider making it accessible only after authentication or allowing access only to certain IP addresses.

Ko Cour
  • 929
  • 2
  • 18
  • 29