1

I'm trying to configure nginx to run as a reverse proxy for two applications: a web frontend (IIS) and a .NET Core backend (Kestrel), all running in a docker swarm. I'm rewriting all calls to /api to the backend and all other calls to the frontend.

In my current setup, everything works fine until I log in to the application. This uses an IdentityServer OAuth/OpenID authentication service, causing an Authorization-header to be added to the request for all calls with a Bearer token. As soon as this header is present, the nginx server returns timeouts from the upstream servers. This happens on both servers, and if I disable passing of the auth header nginx works fine and proxies the request. For the frontend this is not an issue as it does not require the header, but the backend obviously no longer works. The odd thing is if I cut off the header at some point (it is a fairly long string) the request works, but obviously my backend service returns a 500 because it is no longer a valid token.

I've been scratching my head trying to figure out what is wrong and I've tried any number of configuration options. It very much seems like the request is stopped at nginx as neither servers behind the proxy even receive the request when it fails. Logging at the nginx level turns out nothing but 'upstream timed out (110: Operation timed out) while reading response header from upstream'-errors and even increasing that timeout does not do anything, which makes sense as the exact same request without the Authorization header does work.

JDT
  • 143
  • 1
  • 7
  • How long would a correct header be? There may be such a thing as a max header length. – Gerard H. Pille Apr 26 '20 at 18:31
  • The token is 1010 characters long (only letters, numbers, dashes and underscores). If I give another header a similarly long value everything seems to work for that request, so I'm really looking at the Authorization request header as triggering something special in the nginx handling – JDT Apr 26 '20 at 18:51
  • Is Nginx responsible for the authentication? – Gerard H. Pille Apr 26 '20 at 19:23
  • No, nginx should only pass the token around. The backend will take the token and handle everything related to it. – JDT Apr 27 '20 at 06:09
  • Would the backends have trouble reaching the identity server? – Gerard H. Pille Apr 27 '20 at 08:29

1 Answers1

1

The problem is apparently due to the fact that we are running a hybrid swarm with windows and linux nodes. There is a bug related to the network drivers that, oddly enough, few people seem to run into. When the request gets too big the request isn't routed properly inside the docker network. This causes nginx to send out the request and wait for a reply that will never come as the request never makes it to the upstream server...

JDT
  • 143
  • 1
  • 7