Nginx module auth_request waits for either 200 or 401/403 HTTP code from the backend:
The ngx_http_auth_request_module module (1.5.4+) implements client
authorization based on the result of a subrequest. If the subrequest
returns a 2xx response code, the access is allowed. If it returns 401
or 403, the access is denied with the corresponding error code. Any
other response code returned by the subrequest is considered an error.
Your request has a header that require to check for permission:
What I want to do is, if the $header_variable doesn't match a
particular regex I want to return a 403 code.
During auth request/response cycle your backend needs to check the header and return 403 code to prevent next request to happen without authentication. The module auth_request works as a simple test assert which returns one of the code. It does mean backend process that is responsible for this URL to reply http://backendaddress/authorize needs to have an additional check for required header.