2

Just wondering if there is a way to add the "X-Robots-Tag: noindex" tag specifically to PDFs that are served from azure blob storage via an azureedge CDN url. For example -

Blob Storage URL - {subdomain}.blob.core.windows.net/container/test.pdf

CDN URL - {subdomain}.azureedge.net/container/test.pdf

I have read in some places that a way to do this is to add a robots.txt file to the $root container on the blob storage and set "Disallow: *.pdf", however it states that this is not completely reliable and I would rather set the X-Robots-Tag header. Is there a way in azure portal to set it up so that any "*.pdf" file being requested via either the blob or CDN url automatically has this header added to the response?

Many thanks in advance

WombleNumber1
  • 91
  • 7
  • As far as I know, the Azure storage service not provides such functionality to modify response headers. But you can do this in Azure CDN by CDN rule engine:https://learn.microsoft.com/en-us/azure/cdn/cdn-standard-rules-engine-reference – Stanley Gong Mar 09 '21 at 05:49
  • Thanks @StanleyGong this certainly looks like what I am after, however at the moment I cannot see the 'Rules engine' item under 'Settings' when in the CDN endpoint in azure portal, possibly due to the tier that we are using. I am looking into it and will update accordingly. – WombleNumber1 Mar 09 '21 at 15:26

1 Answers1

3

You can find the CDN rule engine here(I am using Standard Microsoft tier):

enter image description here

Test result:

enter image description here

Stanley Gong
  • 11,522
  • 1
  • 8
  • 16
  • Thanks @StanleyGong, that is exactly where I am looking for it, however that option is literally not there for me at the moment. The team are looking into this, and I will mark this as the answer imminently with an update. – WombleNumber1 Mar 10 '21 at 10:20
  • @WombleNumber1, welcome and take your time. Btw, if you can't find that option, maybe you are not using the Standard Microsoft tier. There are multiple tiers of Azure CDN, and they have some feature gaps, details see: https://learn.microsoft.com/en-us/azure/cdn/cdn-features – Stanley Gong Mar 11 '21 at 05:14
  • Indeed @StanleyGong it looks as though from that link the tier that we are using (Standard Akamai) lacks the rules features that the 'Standard Microsoft' tier has, hence I am pretty sure that is why the 'Rules Engine' option is not there for us. Marking this as answer, so many thanks, and will update the original question with information on the resolution for seeing the 'Rules Engine' option once confirmed. – WombleNumber1 Mar 11 '21 at 09:32