Motivation
Suppose you have just woken up and decided that all your services should be running on IPv6 only systems. This is exactly what happened to us and started ipv6onlyhosting.com. Even though you might have a (strong) distaste for IPv4, you have to acknowledge that there are still some users left who don't have IPv6. For that reason, you decide to setup a number of proxies to bridge the IPv4 to the IPv6 world.
This is what happened here at ungleich and this blog article describes one of our proxies and its challenges.
Let's start easy
Like Aldous Huxley said: first things first. Before trying to proxy https, let's try to proxy HTTP first. For this we decided to use a simple nginx configuration that can easily be extended:
map $http_host $proxy_dest {
hostnames;
dashboard.ungleich.ch [2a0a:e5c0:2:12:400:f0ff:fea9:c3c4];
webmail.ungleich.ch [2a0a:e5c0:2:12:400:f0ff:fea9:c3dd];
default [2a0a:e5c0:0:2:400:b3ff:fe39:79ff];
}
server {
listen 80 default_server;
listen [::]:80 default_server;
location / {
root /var/www/html;
try_files $uri @ipv6backend;
}
location @ipv6backend {
proxy_pass http://$proxy_dest:80;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
}
}
This works nicely and the backend, our IPv6 only webserver gets information about who requested originally via the HTTP header X-Forwarded-For. Nice and easy, isn't it?
Let's get cryptic
If http was that easy, https should not be more complicated, right? Wrong. Before going into a solution, let's have a look at the fundamental difference: with http, there is no encryption, no signatures, no trust. So anyone who is in between you and the web server that you want to access can easily setup a proxy and "sniff" your data. This is why the industry moved on to https. For https you need a valid certificate that enables encryption and signing of data so that nobody can see or modify the content in between you and the web server.
Assuming that https is secure, does it mean our proxy project already failed here?
Try 1: Letsencrypt for the rescue?
What is special about our proxy situation? Usually the IPv4 and IPv6 address of a domain name (like www.ungleich.ch) points to the *same* machine. In this case though, we want the IPv6 traffic to go directly to the IPv6 system and the IPv4 traffic to our proxy.
In terms of DNS configuration this means that the AAAA entry points directly to the (real) server, but the A entry points to the proxy.
Our first approach to solve this problem is using letsencrypt, which offers free, automated certificates, so nothing speaks against getting 2 certificates for the same name on different systems.
Well, nothing? Turns out the certbot, which is mainly being used to retrieve certificates, does not allow to select theprotocol (IPv4 vs. IPv6) on which the challenge is expected.
While this is a potential problem, we found a way around this with nginx. Let's review a part of the configuration from above again:
location / {
root /var/www/html;
try_files $uri @ipv6backend;
}
As we don't know how the challenge is being executed, we try to find the challenge locally (using try_files) and if we cannot find it, the
IPv6 host must have requested a challenge and forward it. Nice and easy, isn't it?
This way we can setup a server block for each https backend in nginx:
server {
listen 443;
listen [::]:443;
server_name webmail.ungleich.ch;
access_log /var/log/nginx/access.log;
ssl on;
ssl_certificate /etc/letsencrypt/live/webmail.ungleich.ch/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/webmail.ungleich.ch/privkey.pem;
client_max_body_size 1024m;
location / {
root /var/www/html;
try_files $uri @ipv6backend;
}
location @ipv6backend {
proxy_pass https://$proxy_dest:443;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
}
}
Opening pandora's box
Nice, isn't it? We say yes and no at the same time:
The way we have setup the proxy, we as ungleich could potentially read and even modify requests that come to your server. The person running the IPv6 services enables us to do so by adding the AAAA entry that points to our proxy. For our own systems this does not pose a problem. And for customers?
You can argue that running on any providers infrastructure makes the provider trusted and thus this should not be a problem.
We however take the position that WE DON'T WANT TO SEE YOUR DATA. We are not interested in it and we want to make us even incapable of seeing it.
That is also the reason why we encourage everyone running a web server to redirect all http traffic to https.
Try 2: Change the protocol
Let's stop for a moment and think again on what we want to do. We want to enable the IPv6-only-host to receive http and https traffic
from the IPv4 world. For http proxying it is not a problem, because it is plain text in any case and adding our proxy in between does not
change anything about that. For https the situation became a bit more tricky, as something supposed to be end-to-end encrypted gets split in between.
Instead of looking at layer 7, http/https, let's have a look at IP. So what if we only look at the destination IPv4 address and then forward it to the correct IPv6 host?
Well, it would actually go back to the original reason why IPv6 was created: we would have to have an IPv4 address for every IPv6 address - does not sound very promising, does it?
Try 3: In between
So if breaking up https is too much, dispatching on IP basis is too less, what if we can go in between? It turns out that HA Proxy actually supports SNI (server name indication) based detection of the destination host. The cool thing here is though, instead of proxying it https based, HA Proxy can proxy the connection only using TCP.
This has a couple of advantages: there is no decryption and encryption involved on the proxy and thus there are also no certificates required on the proxy.
A fundamental problem
So is all good? Well, unfortunately using TCP based proxying has a very serious disadvantage: it is impossible to modify the request and thus it is also impossible to add information about which IPv4 address originally sent the request.
So the problem is, either the connection is opened by the proxy and headers can be inserted about the origin or the information about the original request is lost.
Interestingly enough though, this problem does not exist for http in the first place.
Request for comment
Our mission at ungleich is to enable the IPv6 world, because we love IPv6. However this proxy is a bit of a philosophical challenge.
To solve it, we would love to hear your opinion!
- Is it better to open up https and adding the source IP?
- Do you prefer tcp based proxying without any modification?
- There is a better way!
Let us hear your opinion on twitter!
Update #1: using haproxy on the ipv6 only host
We just got notified that haproxy might actually be capable of passing along the source IP using a haproxy based protocol. We will investigate this.
Update #2: IP rewriting to ::ffff:0:a.b.c.d/96
We just received a proposal to simply rewrite the IPv4 address a.b.c.d into to ::ffff:0:a.b.c.d/96 which is specfied for stateless IP/ICMP Translation (SIIT). As far as we can see at the moment, this would be a very interesting way, as the native IPv6 application can directly read out the source IPv4 address. However it requires to implement reverse translation and routing.
Credits for this one go to Gene Redinger, thanks for pointing this out!
Update #3: IP rewriting follow up
While in update #2 we described embedding the source IPv4 address into the known IPv6 prefix ::ffff:0:a.b.c.d/96, there might even be an easier variant available: in all our data centers we already have NAT64 implemented to allow our IPv6 only hosts to reach out to the IPv4 internet. For this to work nicely, we use our own DC specific prefixes.
While this looks very tempting and in theory the http-https-ipv4-ipv6 proxy would "only" need to do layer 7 inspection (either SNI or http host header), there is a fundamental routing problem here: when the IPv6 only VM replies, the reply will not pass through the http proxy and thus will have a generic IPv4 source address that won't match the one of the http proxy and thus establishing a TCP connection will fail.
However, if the NAT64 gateway was actually the same machine as the http proxy, then this approach could work.
We will test this approach in our lab soon!
