ARR mixing HTTP server variables among client users

Question

What I am trying to do?

I am setting up blue-green deployment to achieve zero down-time deployment.

How is the implementation?

1. I have one server.
2. I created a web farm.
3. I set 3 websites blue,green,main at different ports. Main handles all requests from the clients and rewrites requests to web farm.
4. Web farm routes the request to the active website (blue or green) via ARR (Application Request Routing)
5. Since blue/green websites receiving the request from Web Farm/ARR, the REMOTE_ADDR (client ip address) shows 127.0.0.1.
6. I do not want it though it is expected, I have created a URL Rewrite rule that sets the HTTP_X_FORWARDED_FOR_INTERNAL server variable with REMOTE_ADDR at main website.
7. I have created another rule on blue/green websites that sets REMOTE_ADDR server variable with HTTP_X_FORWARDED_FOR_INTERNAL that comes from main website.
8. At result, I am getting the REMOTE_ADDR as real client ip address. YAY!
9. Everything works fine with single client user.

What is not working as expected?

When I connect to the website from different ip addresses, -especially for static contents- blue/green applications receiving wrong ip addresses(REMOTE_ADDR) which are owned by other client.

Example:

1. A and B are different IP addresses.
2. A connects to foo.com/somestaticfile.css.
3. B connects to foo.com/somestaticfile.css.

When I check the IIS Application logs, the log of the request coming from B shows the ip address as 'A' although it should be B!

I can also confirm that request is coming from B according the cookie, we are holding ip address at the cookie, and the ip address at the cookie is not equal to REMOTE_ADDR according to IIS app log.

Edit: I captured another problem, "sometimes" some of the javascript response's mime type is becoming "text/html" in responses, since we implement X-Content-Type-Options=nosniff, those scripts are not loading by browser.

Refused to execute script from 'https://foo.com/bundles/scripts/common?v=Pc3UWD-GF8GEIazC15mnIr_UYtcH3cQPlDPwAf2cXtU1' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.

Edit 2: I have found something useful. When I visit some link that returns 40x status code. The error related with the IP address reproducing every time. And here is the URL that I have being redirected, after I got unauthenticated.

https://foo.com/account/signin?sessionExpired=true&returnUrl=%2ferror%2fnotfound%2f%3f404%3bhttps%3a%2f%2ffoo.com%3a10003%2ferror%2fnotfound

The decoded value of the returnUrl is: "/error/notfound/?404;https://foo.com:10003/error/notfound"

Weird?

What I have tried to solve the situation?

I thought it is some kind of caching issue then I disabled all the caching mechanisms I know which are;

1. Disabled IIS Cache from global settings.
2. Disabled ARR Cache via creating cache control rule.
3. Disabled Web Farm Cache.

It did not work though.

I have cleared all the configurations and used ARR Helper which is developed for forwarding IP address to hindmost web application in configurations like this, I encountered with the same problem.

Configuration Files You May Need

Rule that sets HTTP_X_FORWARDED_FOR_INTERNAL in main web app.

<rule name="Set HTTP_X_FORWARDED_FOR_INTERNAL">
                <match url="(.*)" />
                <conditions logicalGrouping="MatchAll" trackAllCaptures="false" />
                <serverVariables>
                    <set name="HTTP_X_FORWARDED_FOR_INTERNAL" value="{REMOTE_ADDR}" />
                </serverVariables>
                <action type="None" />
            </rule>

Rule that sets REMOTE_ADDR in blue/green apps.

<rule name="Set REMOTE_ADDR">
      <match url="(.*)" />
      <conditions logicalGrouping="MatchAll" trackAllCaptures="false">
        <add input="{HTTP_X_FORWARDED_FOR_INTERNAL}" pattern="^$" negate="true" />
      </conditions>
      <serverVariables>
        <set name="REMOTE_ADDR" value="{HTTP_X_FORWARDED_FOR_INTERNAL}" />
        <set name="REMOTE_HOST" value="{HTTP_X_FORWARDED_FOR_INTERNAL}" />
      </serverVariables>
      <action type="None" />
    </rule>

Rule that disabled ARR caching at the global level.

 <rewrite>
        <globalRules>
            <rule name="ARR_CacheControl_9dc69ea7-26f6-4654-af58-2e289d681463" enabled="true" patternSyntax="Wildcard" stopProcessing="true">
                <match url="*" />
                <serverVariables>
                    <set name="ARR_CACHE_CONTROL_OVERRIDE" value="1,no-cache" />
                </serverVariables>
                <action type="None" />
            </rule>
        </globalRules>
    </rewrite>

Whole web farm configuration

<webFarms>
    <webFarm name="iissucks" enabled="true">
        <server address="bluecloud" enabled="true">
            <applicationRequestRouting httpPort="880" httpsPort="8443" />
        </server>
        <server address="greencloud" enabled="true">
            <applicationRequestRouting httpPort="980" httpsPort="9443" />
        </server>
        <applicationRequestRouting>
            <healthCheck url="https://iissucks/account/signin" interval="00:00:01" statusCodeMatch="200-599" responseMatch="I_am_healthy." />
            <protocol xForwardedForHeaderName="X-Forwarded-For-Farm" includePortInXForwardedFor="false">
                <cache enabled="false" queryStringHandling="NoCaching" validationInterval="00:00:00" />
            </protocol>
        </applicationRequestRouting>
    </webFarm>
    <applicationRequestRouting>
        <hostAffinityProviderList>
            <add name="Microsoft.Web.Arr.HostNameRoundRobin" />
        </hostAffinityProviderList>
    </applicationRequestRouting>
</webFarms>

Answer 1

I'm far from being an IIS expert, but I found your problem interesting. I see that your global cache disabling rule has a stopProcessing="true" parameter set in it. Maybe your variable setting rules just aren't being executed after the cache disable rule stops processing any further rules.