Without being able to see all that is in your configuration file, it still appears clear that the issue you are having arises because you have preemptively denied "all" users access to the error page.
In other words, you can't send back an error page that is itself protected by the basic authorization. The server doesn't know how to handle this recursive situation properly, and so it doesn't.
Make sure that you explicitly override and disable the .htaccess-requested authorization for the directory which contains your custom error page(s).
For example, if your error-file directory is /error/ as you have indicated:
<Directory /error/>
Order allow,deny
Allow from all
</Directory>
@Fred-gant
To answer your question, let me expand a little bit on the 401 process. It helps to understand that HTTP is not a stateful protocol. In other words, every request and every response exist in isolation from subsequent requests and responses (with respect to the actual HTTP protocol--of course, both your browser and the server can maintain stateful information about each request, and attempt to associate a series of requests with one another).
- In your server configuration you define an area on your server that is protected, and you define the credentials which enable access.
- When any object within the protected area is first requested, the server responds with a 401 header, a
WWW-Authenticate header and authorization challenge, and optional content. The request and response process is complete.
- The browser receives the completed response, parses the authorization challenge sent in the server's response, and presents it to you in human-readable format.
- If you choose to cancel the authorization process, then the browser presents you with the response body (if any).
- If you choose to supply your login credentials, the browser encodes them in whatever format requested by the server, and begins a new request.
- When any object within the protected area is requested via this second request, the server validates or invalidates the credentials that are supplied, and either returns the object requested, or else returns the same 401 header and content as before.
- If a browser receives the same 401 header/content after you attempted to supply your credentials, then you are either requested for credentials again, or you are presented with the response body sent with the 401 header from the server.
- If the server validated the credentials supplied and returns the requested object, then your browser will continue sending the same credentials with each request until the current browser session is terminated. Let me call this the "post-authentication mode" of your browser.
The most important thing to note is that a complete response is received from the server before you even see the request from login credentials from your browser.
And the only way that your browser even realizes that it needs to present you with a request for login credentials is if there is not only a 401 response header, but also a WWW-Authenticate header and challenge as well.
If your 401 custom error document causes your browser to display the error page rather than requesting login credentials, then it must be that the authentication header and challenge, or perhaps even the 401 status code itself, are not configured to accompany the 401 error document.
To us that sounds crazy, but to the server's non-human logic, even if no authorization challenge is ever sent, it still makes perfect sense to send out an error document to all non-authorized requests, and the actual object to all browsers in "post-authentication mode."
You can find out if this is what is happening by commenting out the custom 401 error file directive, and then logging in with your browser, loading a protected page, and then re-enabling the custom 401 error file directive, and then refreshing the protected page in your browser that is already in the "post-authentication mode."
What I expect is that your pre-authorized browser will continue sending its credentials with this request, and that the server will respond with the requested protected object.
So all authorized requests are working properly as always, but it is just that the 401 error document and/or headers do not contain the authentication challenge trigger that tells your browser that it needs to ask you for your credentials.
