Need to block subdomain using robots.txt which is on same directory level

Question

I have one problem

I have domain name for example www.testing.com and new.testing.com so i do not want to new.testing.com display in any search engine. I have added one robots.txt to the new.testing.com. And both site has same parent directory

--httpdoc
----testing.com
----new.testing.com

So i want to know that can i handle both site using one robots.txt of testing.com is it possible???

Please suggest me solution if possible.

score 10 · Accepted Answer · answered Sep 19 '14 at 08:21

10

The best thing you could do is to add separate robots.txt files. Put one in each directory. You should have

testing.com/robots.txt and
new.testing.com/robots.txt

After adding robots.txt file to new.testing.com, you should add following code to keep search engines away.

User-agent: *
Disallow: /

answered Sep 19 '14 at 08:21

bmiljevic

742
8
16

2

if domain and subdomain same directory how can disallow only subdomain? disallow: /new.domain.com/ that true? or have any another idea? please help... – gokhan Dec 17 '14 at 17:22

score 6 · Answer 2 · answered Jun 02 '15 at 14:53

6

RewriteEngine on
RewriteCond %{HTTP_HOST} ^subdomain.website.com$
RewriteRule ^robots\.txt$ robots-subdomain.txt

Then add the following to /robots-subdomain.txt:

User-agent: *
Disallow: /

answered Jun 02 '15 at 14:53

DD.

21,498
52
157
246

1

Exactly what i was looking for. Thanks! – ANDiTKO Mar 01 '17 at 11:01

score 0 · Answer 3 · answered Jul 09 '18 at 13:34

0

Following rules inside works for me,

RewriteEngine On
RewriteCond %{HTTP_HOST} ^subdomain\.maindomain\.com$ [NC]
RewriteRule ^/robots.txt$ /nobots.txt [L]

And add 'nobots.txt' to root directory as follows,

User-agent: *
Disallow: /

answered Jul 09 '18 at 13:34

Hemant Thorat

2,386
20
14

Need to block subdomain using robots.txt which is on same directory level

3 Answers3