Time for some champagne, because WE ARE MULTIHOMED! After working closely with engineers from Init7 and Netstream, we are now online with 2 connections! With this blog post we would like to send our huge thanks to everyone for the great help. And as you have guessed, again 100% opensource thanks to The Bird from cz.nic.
Motivation
When we started data center light, we started small. One router, 3 servers, 2 switches, 1 uplink. Over the years it developed into multiple locations, many routers, even more servers and ... who is counting switches anyway?
More important though, over the years we faced some downtimes due to outages on our uplink. After negotiating and talking to a variety of ISPs, we found a provider for our second link. But now we are faced with questions like how to route what from and to where? Which routes to announce at which uplink?
So while the objective is to have higher reliability, we also gain some complexity.
How to multi home: try 1
In our first prototype, we tried to setup BGP on our switches. We received the full routing table with about 730k IPv4 routes and about 52k IPv6 routes. Our switches were not made for this amount of routes and they choose random routes to store in the FIB, which made them to drop our own prefixes. It took us some time to realise this problem and while we could have introduced filtering on the switches, we choose a different way.
Multi home with BIRD
That’s right, with the bird!
You might have heard of the BIRD Internet Routing Daemon (yes, another recursive acronym!), a friendly open source routing suite. What is great about is that it is very simple to use and that it runs on regular Linux/BSD systems. Closed systems like most switches are, in our opinion, not state of the art anymore.
Our setup
We have 2 routers in each data center, 4 routers in total. While OSPF is nicely designed for an intra domain routing, we decided to go with iBGP and eBGP due to its simplicity. The fact that we need a full mesh with BGP does not matter for us, as our routers are fully configured by cdist., so it is only 1 configuration for all routers. (By the way, cdist recently had a really nice makeover of its website, clean and elegant! If you haven't yet, check it out.)
Customer ASs now supported
Now that we are running our own routing in our own AS (autonomous system), we also support customers bringing their own IPv6 networks. And it turns out, the first customer AS is already online!
More on multi home will follow!
So that’s it for today, but next up we’ll tell you more fun details. Topics like how our configuration exactly looks like, and how to use bird with four routers will be explained in our next blog article - so stay tuned! You can follow us on Twitter @ungleich to keep track of our latest blog release.
If you want to bring your own IPv6 network, get a VM at IPv6onlyhosting or Data Center Light.
