Get X-robots-tag in PHP

Asked Dec 04 '18 at 14:18

Active Dec 04 '18 at 14:23

Viewed 289 times

I would like to check if a list of URL doesn't have a noindex. So I check already

< meta name="robots" >

with DOM, but I would like to also check the X-robots-tag. Is there an easy way to do it?

The only way I can imagine is a foreach() on get_headers($url).

edited Dec 04 '18 at 14:23

Blackbam

asked Dec 04 '18 at 14:18

jd440

I'm not familiar with the `x-robots-tag` but maybe `get_headers($url, 1)` and `empty($var['X-robots-tag')`.. assuming any value means don't crawl (`0` would be valid here there) – user3783243 Dec 04 '18 at 14:22

0 Answers0