Asp.Net Core, reverse proxy and X-Forwarded-* headers

Running microservices and applications using Asp.Net Core and Kestrel inside docker on Linux fronted by one or several reverse proxies will create a few issues that has to be addressed.

A typical scenario is one proxy acting ingress controller to the container orchestration platform and sometimes a second reverse proxy for internet exposure.

If we ‘do nothing’ .Net Core will consider the last proxy as ‘the client’ making the requests. Framework components or custom code using the HttpContext to get information on the original client will get the wrong information.

One example where this causes problems are the OpenIdConnect middleware which will generate wrong redirects links. Another is when we want to log the client IP-address.

Luckily the http headers X-Forwarded-* are here to address this exact problem and .Net Core has great support with the ForwardedHeaders middleware. Each reverse proxy will add to the X-Forwarded headers and the middleware will change the HttpContext accordingly.

With the ForwardedHeaders middleware configured with XForwardedHost + XForwardedProto (which is all that is needed for a OIDC redirect) it work fine.

//ConfigureServices
services.Configure<ForwardedHeadersOptions>(options =>
{
    options.ForwardedHeaders =
       //This one did not work ForwardedHeaders.XForwardedFor | 
       ForwardedHeaders.XForwardedHost | 
       ForwardedHeaders.XForwardedProto;
});

//Add to pipeline in Configure
app.UseForwardedHeaders();


I had the middleware was configured like above. Lets see if we can find out why it did not work in Docker as soon as i added X-Forwarded-For.

Create a test-rig

First lets create a simple test-rig so that we can simulate one or more proxies in a simple way.

Simply output both the HttpContext based properties and the relevant http header values.

Now lets make a request ‘faking’ proxies adding the X-Forwarded-* headers. And lets make one with and one without the X-Forwarded-For header to the service running in Docker

I use the VScode extension REST-Client here but Postman would also work.

The application now act as if i the request was made to https://superapp.com and not http://localhost:99

Now if we add the ForwardedHeaders.XForwardedFor to the middleware configuration and give it another try things change – suddenly it does not work anymore. The same request to Kestrel running directly on Windows work fine.

Now the middleware does not update the HttpContext based on the X-ForwardedFor-* headers. Unlike some other settings such as header symmetry errors if RequireHeaderSymmetry is enabled which will generate a warning, this is completely silent in the logs.

The solution is very simple. The application running inside docker is on a different network than the ‘proxy’ and the middleware default behavior is to only trust proxies on loopback interface. Lets fix that.

//ConfigureServices
services.Configure<ForwardedHeadersOptions>(options =>
{
    options.ForwardedHeaders =
       ForwardedHeaders.XForwardedFor | 
       ForwardedHeaders.XForwardedHost | 
       ForwardedHeaders.XForwardedProto;

       options.ForwardLimit=2;  //Limit number of proxy hops trusted
       options.KnownNetworks.Clear();
       options.KnownProxies.Clear();
});

If we clear the default list of KnownNetworks and KnownProxies it work fine. In a real scenario we would add the real addresses to
KnownNetworks and KnownProxies based on a configuration property so that it could change without changing code.

Now all three properties we are interested work as expected! The originating client IP is based on X-Forwarded-For header.

.Asp.Net Core 3.x

Asp.Net Core 3.x have simplified the use of ForwardedHeaders middleware. If we set the environment variable ‘ASPNETCORE_FORWARDEDHEADERS_ENABLED’ to true the framework will insert the ForwardedHeaders middleware automatically for us.

Lets try it!

docker run -e ASPNETCORE_FORWARDEDHEADERS_ENABLED=true -p 99:80 forwardedheaderswebtester


We now no longer have the correct host and our OIDC middleware will NOT work as expected. Why is that?

Well, lets have a look at the Asp.Net Core sourcecode as to what default behavior is when that environment variable is used.
https://github.com/dotnet/aspnetcore/blob/1480b998660d2f77d0605376eefab6a83474ce07/src/DefaultBuilder/src/WebHost.cs#L244

In the code we can see that it adds XForwardedFor + XForwardedProto. It does NOT add XForwardedHost.

So in my scenario where i need all three headers i will have to configure the middleware manually as per above and everything work as expected!

The code above is available here :
https://github.com/magohl/ForwardedHeadersWebTester

SoapUI in high DPI

Running SoapUI on my new fantastic 4K 15.6″ Windows 10 laptop had me looking for a pair of binoculars.

Windows 10 itself handles the extreme high DPI resolutions quite good. But SOAPUI does not scale very well. Luckily the solution is simple.

Resolution

Add the following registry setting:

reg add HKLM\Software\Microsoft\Windows\CurrentVersion\SideBySide /v PreferExternalManifest /d 1 /t REG_DWORD

Create a file called {your-soapui-exe}.manifest in the SoapUI bin folder, such as soapUI-5.2.1.exe.manifest, with the following content:

<?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes”?> <assembly xmlns=”urn:schemas-microsoft-com:asm.v1″ manifestVersion=”1.0″ xmlns:asmv3=”urn:schemas-microsoft-com:asm.v3″> <description>soapui</description> <asmv3:application> <asmv3:windowsSettings xmlns=”http://schemas.microsoft.com/SMI/2005/WindowsSettings”> <ms_windowsSettings:dpiAware xmlns:ms_windowsSettings=”http://schemas.microsoft.com/SMI/2005/WindowsSettings”>false</ms_windowsSettings:dpiAware> </asmv3:windowsSettings> </asmv3:application> </assembly>

 

Done. Restart SoapUI and it scales so much better!

This could also be applied to other applications having the same problem.

WCF service hosted in Azure Websites

While WCF might not be the most viable .NET technology stack on the open web as opposed to WEB API 2.0 it is still very relevant in enterprise and B2B scenarios.

It is sometimes considered hard to configure and host WCF. Lets see how hard it really is now with .NET 4.5.

The other day i quickly needed a simple test SOAP endpoint exposed on the internet. I thought i would host it in Azure Websites.

First let us create the web site in Azure. As an alternative to normal ftp-deploy lets use Azure Websites great Kudu features and GIT support.

C:\>mkdir EchoService
C:\>cd EchoService
C:\EchoService>azure site create EchoService --git --location "North Europe"

 

Lets crate a simple untyped WCF service echoing any SOAP-request back as a response. This is probably not a real world scenario although the untyped nature of System.ServiceModel.Channels.Message is really powerful. But any WCF service you find appropriate would work.

The sample below is a minimal single-file wcf service without any code-behind. Save it to a file called EchoService.svc.

<%@ ServiceHost Language="C#" Service="EchoService" %>

using System.ServiceModel;
using System.ServiceModel.Channels;

[ServiceContract]
public class EchoService
{
    [OperationContract(Action="*", ReplyAction="*")]
    public Message Echo(Message message)
    {
        return message;
    }
}

 

Save the file and push it to the remote repository automatically created in your Azure Website.

c:\EchoService>git add .
c:\EchoService>git commit -m"First checkin"
c:\EchoService>git push azure master

 

Done!

GIT pushed the EchoService.svc file to the remote repository and Kudu automatically deployed it into the website wwwroot folder. If you want to learn more on the amazing kudu stuff in Azure Websites i highly recommend having a look at the short videos made by Scott Hanselman and David Ebbo.

You can reach the service at http://yourwebsitename.azurewebsites.net/EchoService.svc and maybe use something like SoapUI to try it out. The WCF default configuration will expose an endpoint using the BasicHttpBinding meaning any SOAP 1.1 envelope will work. Metadata publication is disabled by default but as this is an untyped service there is really no need for it. If needed it can easily be enabled in code or configuration.

soapui_echoservice

As show Microsoft PaaS service Azure Websites is i really simple way to host a WCF endpoint in the cloud. With the help of .NET 4.5 this is easier than ever.