strange 401 error appears for some urls when using .htaccess to redirect http to https

Question

OK, here is the 7th day of unsuccessfull attempt to find an answer why 401 error appears...

Now, .htaccess in the root folder contains the only 3 strings (was simplified) and there are NO more .htaccess files in the project:

RewriteEngine On
RewriteCond %{HTTPS} !on
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

So, it redirects all requests to be https. It works fine for any urls, even for /administration directory.

So,

http://mydomain.com

becomes

https://mydomain.com

If https://mydomain.com was entered, there are no redirections.

http://mydomain.com/administration/index.php

becomes

https://mydomain.com/administration/index.php

If https://mydomain.com/administration/index.php was entered, there are no redirections.

That's clear, and the problem is below.

I want /administration directory to be password protected. My Shared Hosting Control Panel allows to protect directories without manual creating of .htaccess and .htpasswd (you choose a directory to protect, create username and password, and .htaccess and .htpasswd are created automatically). So, .htaccess appears in the /administration folder. .htpasswd appears somewhere else, the path to .htpasswd is correct, and everything looks correct (it works the same way as to create it manually). So, there are 2 .htaccess files in the project, one in the root directory and one in the /administration directory (with .htpasswd at the directory .htaccess knows where it is).

Once the password is created, the results are:

You enter:

https://mydomain.com/administration/index.php

Then it asks to enter a password. If you enter it correctly, https://mydomain.com/administration/index.php is displayed. The result: works perfect.

But, if you enter http://mydomain.com/administration/index.php (yes, http, without S) then instead of redirecting to the same,but https page, it redirects to

https://mydomain.com/401.shtml (starts with httpS)

by unknown reason and even does NOT ask a password. Why?

I've contacted a customer support regarding this question and they are sure the problem is in .htaccess file, and they do not fix .htaccess files (that's clear, they do not, I don't mind).

Why does this happen? Did I forget to put some flags, or some options to change default settings in the .htaccess file?

P.S.Creating .htaccess and .htpasswd manually (not from hosting Control Panel) for the folder /administration causes the same 401 error in case if not https, but http was entered.

And the problem appears with URLs to /administration directory only.

Thank you.

Answer 1

Try using this instead. Not the L and R flag.

RewriteEngine On
RewriteCond %{HTTPS} !on
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

Also clear your browsers cache first, to remove the old incorrect redirect.

If that doesn't work try using this.

RewriteCond %{HTTPS} !on
RewriteCond %{THE_REQUEST} ^(GET|HEAD)\ ([^\ ]+)
RewriteRule ^ https://%{HTTP_HOST}%2 [L,R=301]

I feel a bit bad about writing it, as it seems kind of hackish in my view.

EDIT Seems the 2nd option fixed the problem. So here is the explanation as to why it works.

The authentication module is executed before the rewrite module. Because the username and password is not send when first requesting the page, the authentication module internally 'rewrites' the request url to the 401 page's url. After this mod_rewrite comes and %{THE_REQUEST} now contains 401.shtml instead of the original url. So the resulting redirect contains the 401.shtml, and not the url you want.

The get to the original (not 'rewritten') url, you need to extract it from %{THE_REQUEST}. THE_REQUEST is in the form [requestmethod] [url] HTTP[versionnumber]. The RewriteCond extracts just the middle part ([url]).

For completeness I added the [L,R=301] flags to the second solution.

Answer 2

I think I found an even better solution to this!

Just add this to your .htaccess

ErrorDocument 401 "Unauthorized"

Solution found at:

http://forum.kohanaframework.org/discussion/8934/solved-for-reall-this-time-p-htaccess-folder-password-protection/

-- EDIT

I eventually found the root cause of the issue was ModSecurity flagging my POST data (script and iframe tags cause issues). It would try to return a 401/403 but couldn't find the default error document because ModSecurity had made my htaccess go haywire.

Using ErrorDocument 401 "Unauthorized" bypassed the missing error document problem but did nothing to address the root cause.

For this I ended up using javascript to add 'salt' to anything which was neither whitespace nor a word character...

  $("form").submit(function(event) {
    $("textarea,[type=text]").each(function() {
      $(this).val($(this).val().replace(/([^\s\w])/g, "foobar$1salt"));
    });
  });

then PHP to strip the salt again...

function stripSalt($value) {
  if (is_array($value)) $value = array_map('stripSalt', $value);
  else $value = preg_replace("/(?:foobar)+(.)(?:salt)+/", "$1", $value);

  return $value;
}
$_POST = stripSalt($_POST);

Very, Very, Very Important Note:
Do not use "foobar$1salt" otherwise this post has just shown hackers how to bypass your ModSecurity!

Regex Notes:
I thought it may be worth mentioning what's going on here...

(?:foobar)+ = match first half of salt one or more times but don't store this as a matched group;

(.) = match any character and store this as the first and only group (accessible via $1);

(?:salt)+ = match second half of salt one or more times but don't store this as a matched group.

It's important to match the salt multiple times per character because if you've hit submit and then you use the back button you will go back to the form with all the salt still in there. Hit submit again and more salt gets added. This can happen again and again until you end up with something like: foobarfoobarfoobarfoobar>saltsaltsaltsalt

Answer 3

I was not satisfied with the solutions above so I came up with another one:

In a modern web server configuration we should redirect all traffic to HTTPS so the user can not reach any content without HTTPS. After the user's browsing our content with HTTPS we can use authentication. With this in mind we can wrap the authentication directive in an If directive:

<If "%{HTTPS} == 'on'">
  AuthType Basic
  ...
</If>

You can leave and use Rewrite directives as you like.

With this solution:

• you must not change ErrorDocument as suggested by Hoogs
• you must not extract path from THE_REQUEST in a hackish way as suggested by Gerben

Answer 4

This is the type of thing is that is a bit tricky to troubleshoot on Apache without the box right in front of you, but I what I think is happening is that your rewrite directive is being processed after path resolution, and it's the path resolution that has the password requirement.

Backing up a bit, the way a URL is resolved in Apache is that the request comes in and gets handed from module to module, kind of like a bucket brigade. Each module does its own thing....some modules do content negotiation, some translate URLs to file paths, some check authentication, one of them is mod_rewrite ...

One place where you see this in the configuration is actually that there is both a Location directive and a Directory directive which seem the same in most respects, but they are different because Locations talk about URLs and Directories talk about filesystem paths.

Anyhow, my guess is that going down the bucket brigade, Apache figures out that it needs a password to access that content before it figures out that it needs to redirect to HTTPS. (mod_rewrite is kind of a crazy module and it can mess with all kinds of things in surprising ways..it can do path translation, bits and pieces of rewrite, make subrequests, and a bunch of other nutty things).

There are few ways you can fix this that I can think of.

1. Change your directory root in the vhosts container for the http site so that it can't find the passworded file (this would be my approach)
2. Change your module load order so that mod_rewrite happens earlier in the chain (may have unexpected consequences)
3. Use setenvif

That last one needs more explanation. Remember the bucket brigade I told you about? Apache modules can also set environment variables, which are completely outside of the module->module->module->chain. You could, perhaps, set an environment variable if the site is not HTTPS. Then however you set up your access control could use the SetEnvIf directive to always allow access to the resource if it's set, BUT you have to make sure for sure that you're going to hit that rewrite rule.

As I said, my choice would #1 but sometimes people need to do crazy things, and Apache will let you.

My real-world SOP for https:// sites these days is that I just shoot all of my port 80 content over to a single vhost that can't serve any content at all. Then i mod_rewrite everything over https://... badda bing, badda boom, no http and no convoluted security risks.