Asp.Net Core, reverse proxy and X-Forwarded-* headers

Uncategorized

| February 24, 2020 | 4

Running microservices and applications using Asp.Net Core and Kestrel inside docker on Linux fronted by one or several reverse proxies will create a few issues that has to be addressed.

A typical scenario is one proxy acting ingress controller to the container orchestration platform and sometimes a second reverse proxy for internet exposure.

If we ‘do nothing’ .Net Core will consider the last proxy as ‘the client’ making the requests. Framework components or custom code using the HttpContext to get information on the original client will get the wrong information.

One example where this causes problems are the OpenIdConnect middleware which will generate wrong redirects links. Another is when we want to log the client IP-address.

Luckily the http headers X-Forwarded-* are here to address this exact problem and .Net Core has great support with the ForwardedHeaders middleware. Each reverse proxy will add to the X-Forwarded headers and the middleware will change the HttpContext accordingly.

With the ForwardedHeaders middleware configured with XForwardedHost + XForwardedProto (which is all that is needed for a OIDC redirect) it work fine.

//ConfigureServices
services.Configure<ForwardedHeadersOptions>(options =>
{
    options.ForwardedHeaders =
       //This one did not work ForwardedHeaders.XForwardedFor | 
       ForwardedHeaders.XForwardedHost | 
       ForwardedHeaders.XForwardedProto;
});

//Add to pipeline in Configure
app.UseForwardedHeaders();

I had the middleware was configured like above. Lets see if we can find out why it did not work in Docker as soon as i added X-Forwarded-For.

Create a test-rig

First lets create a simple test-rig so that we can simulate one or more proxies in a simple way.

Simply output both the HttpContext based properties and the relevant http header values.

Now lets make a request ‘faking’ proxies adding the X-Forwarded-* headers. And lets make one with and one without the X-Forwarded-For header to the service running in Docker

I use the VScode extension REST-Client here but Postman would also work.

The application now act as if i the request was made to https://superapp.com and not http://localhost:99

Now if we add the ForwardedHeaders.XForwardedFor to the middleware configuration and give it another try things change – suddenly it does not work anymore. The same request to Kestrel running directly on Windows work fine.

Now the middleware does not update the HttpContext based on the X-ForwardedFor-* headers. Unlike some other settings such as header symmetry errors if RequireHeaderSymmetry is enabled which will generate a warning, this is completely silent in the logs.

The solution is very simple. The application running inside docker is on a different network than the ‘proxy’ and the middleware default behavior is to only trust proxies on loopback interface. Lets fix that.

//ConfigureServices
services.Configure<ForwardedHeadersOptions>(options =>
{
    options.ForwardedHeaders =
       ForwardedHeaders.XForwardedFor | 
       ForwardedHeaders.XForwardedHost | 
       ForwardedHeaders.XForwardedProto;

       options.ForwardLimit=2;  //Limit number of proxy hops trusted
       options.KnownNetworks.Clear();
       options.KnownProxies.Clear();
});

If we clear the default list of KnownNetworks and KnownProxies it work fine. In a real scenario we would add the real addresses to
KnownNetworks and KnownProxies based on a configuration property so that it could change without changing code.

Now all three properties we are interested work as expected! The originating client IP is based on X-Forwarded-For header.

.Asp.Net Core 3.x

Asp.Net Core 3.x have simplified the use of ForwardedHeaders middleware. If we set the environment variable ‘ASPNETCORE_FORWARDEDHEADERS_ENABLED’ to true the framework will insert the ForwardedHeaders middleware automatically for us.

Lets try it!

docker run -e ASPNETCORE_FORWARDEDHEADERS_ENABLED=true -p 99:80 forwardedheaderswebtester

We now no longer have the correct host and our OIDC middleware will NOT work as expected. Why is that?

Well, lets have a look at the Asp.Net Core sourcecode as to what default behavior is when that environment variable is used.
https://github.com/dotnet/aspnetcore/blob/1480b998660d2f77d0605376eefab6a83474ce07/src/DefaultBuilder/src/WebHost.cs#L244

In the code we can see that it adds XForwardedFor + XForwardedProto. It does NOT add XForwardedHost.

So in my scenario where i need all three headers i will have to configure the middleware manually as per above and everything work as expected!

The code above is available here :
https://github.com/magohl/ForwardedHeadersWebTester