"Not Found: /406.shtml" from django

Question

I'm running django with apache fcgi on a shared host. I've set it up to report 404 errors and keep seeing Not Found: /406.shtml via emails (I'm guessing the s is because it's https only). However I have error documents already set up in .htaccess:

ErrorDocument 406 /error/406.html

I was getting a bunch of similar 404 errors from django before setting up an ErrorDocument for each one, but it's still happening for 406. From a grep 406 through the apache error log I'm seeing an occasional 406 (not 404) error for 406.shtml, such as the following, but not nearly as often as django emails me:

[Fri ...] [error] [client ...]
    ModSecurity: Access denied with code 406 (phase 1).
    Pattern match "Mozilla ... AhrefsBot ...)" at REQUEST_HEADERS:User-Agent.
    [file "/usr/local/apache/conf/mod_sec/mod_sec.hg.conf"] [line "126"]
    [id "900165"] 
    [msg "AhrefsBot BOT Request"]
    [hostname "www.myhostname.com"]
    [uri "/406.shtml"]
    [unique_id "..."]

I'm not even sure if this is apache redirecting internally to 406.shtml and it being forwarded on to django or if some bot is trying to find 406.shtml directly. The former seems to indicate a problem with ErrorDocument. The latter isn't really my problem, but then either I should be seeing a 404 for 406.shtml in the apache logs or nothing at all because django will handle the 404? How can I track it down further?

I haven't been able to reproduce the issue just by visiting my site, but I'd like to know what's going on.

Answer 1

You have ModSecurity installed in your Apache which is a WAF which attempts to protect your website from attacks, bots and the like. These, like email spam are part and parcel of running a website now a days unfortunately.

ModSecurity is an add on module to Apache which allows you to define rules and then it runs each request against those rules and decides whether to block the request or not.

In this case a rule (900165, which is defined in file "/usr/local/apache/conf/mod_sec/mod_sec.hg.con) has decided to block this request with a 406 status based on the user agent (AhrefsBot).

Ahref is a website which crawls the web trying to build up a database of links. It's used by SEO people to see who links to your websites (back links are very important to SEO) as Google (who you think would be better providers of this type of information) only give samples of links rather than full listing.

Is AhrefBot a danger and should it be blocked? Well that's a matter of opinion. Assuming it's really AhrefBot (some nefarious bots might pretend to be it so as to look legitimate so check the IP address to see the hostname it came from), then it's probably wasting your resources without doing you much good. On the other hand this is the price of an open web. Your website is available to the public and so also to those that write bots and tools (good or bad).

Why does it return a 406? Well that's how your ModSecurity and/or your rule is defined. Check your Apache config. 406 is a little unusual as would normally expect a 403 (access denied) or 500 (internal server error).

What's the 406.shtml file? That I don't get. A .shtml is a HTML file which also allows server side includes to embed other files and code into an HTML file. They are not used much any more to be honest as the likes of PHP and/or other languages are more common. It could be an attack: I.e. someone's attempting to upload the 406.shtml file and then cause it to be called so it "executes" and includes the contents of the file, potentially giving access to files Apache can see which are not available on the webserver, or the user has requested that (for some reason) or Apache is configured to show that for 406 errors or the ModSecurity rule is redirecting to that file.

Hopefully that gives a good bit of background, and best thing I can suggest is to go through your Apache config file, and any other config files it loads (including mod_sec.hg.con file which it must load) to fully understand your set up and the. Decide if you need to do anything here.

You could do one of several things:

1. Leave as is. ModSecurity is doing what it was told to do and blocking this with a 406
2. Turn off this rule and allow AhrefRef through so you don't get alerted by this.
3. Alter the ModSecurity config/rule to return an error other than 406 so you can ignore it
4. Turn off ModSecurity completely. I think it is a good tool and worthwhile but does take some time and effort to get most out of it.
5. Set up the 406 error page properly. To do that you need to understand why it's attempting to return 406.shtml at the moment.

Also not sure which of these options are available to you as you are on a shared host and might not have full access. If so speak to your hosting provider for advice.