Arjan Schaaf bio photo

Arjan Schaaf

Linux, AWS, Azure, Docker & Java rock my world. But far more important: I'm a husband and father of two. Love to BBQ and I'm famous in my world for my burgers & ribs :-)

Twitter LinkedIn Github

This blog is long overdue.. I’m meant to come back sooner with new data on the artificial performance tests I did in 2015 with the different Software Defined Network(SDN) solutions available in the Docker ecosystem. Back then libnetwork was just announced but not yet generally available. With the release of Docker 1.9.0, libnetwork became part of the standard Docker releases.

So why is libnetwork important?

Well first of all: we have a new implementation available: the default driver that Docker ships with libnetwork. This default driver is based on the work started at SocketPlane http://socketplane.io/.

SocketPlane got acquired by Docker in 2015 and is now the default driver available when you start using Docker libnetwork.

Secondly: the promise of libnetwork is “batteries included but swappable”. Other SDN providers like Weave and Project Calico implemented support for libnetwork in their solutions. So you don’t like the default driver? Swap it with another provider, while using the network setup introduced by Docker libnetwork, without the need to depend on a vendor specific approach. Which is what you needed to do when you started using Flannel, Weave or Project Calico before the introduction.

All the SDN solutions I tested previously (Project Calico and Weave) provide libnetwork support except for CoreOS Flannel.

What are we testing?

Not every SDN performance test is equal. It is hard to create reproducible test results and what might work in one setup will not work in another (more on that later). So I’ve chosen a setup that is based on the setup we use for most of our production environments.

I’ve could have made different choices, which would have resulted in a different outcome. If you want to be sure what test results you would get on your own environment: please test it yourself. Your milage may vary. So we are testing on Amazon Web Services, using the latest m4 General Purpose family. This is an important detail, because the modern AWS instance types support Jumbo Frames with an MTU of 9001 instead of the old default of 1500.

For more information about the use of Jumbo frames on AWS: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html

Using the maximum packet size based on the MTU of your machine can make a huge impact on the test results. We use CoreOS (beta build 877.1.0) as the Linux distribution to run the Docker containers on. CoreOS 877.1.0 is shipped with Docker 1.9.1 and Kernel 4.3.0. Both these version numbers turned out to be very important for the test results…

So with this Docker distribution one can easily create a docker network to test the network performance using the different drivers. In this blog I’ll compare the libnetwork implementation provided by Docker and Weave. Other implementations and configurations will follow in upcoming blogs. For reference we will compare these results with the non-libnetwork Flannel SDN (vxlan based) and connecting the test server & client containers over the host interfaces of the EC2 vm’s. That last setup bypasses the Docker SDN completely and should give more insight into the maximum performance without the overhead of a SDN.

And we use both the iperf3 and qperf test tools to test the performance. Qperf is a very simple tool that does short tests runs, I use iperf3 to do a longer test run, using more parallel connections. The higher number of parallel connections causes more overhead, but is a (more?) realistic test in my opinion.

The results on two m4.xlarge instances in one EC2 availability zone

MB/s vs SDN solutions

What do these results tell us?

Looking at the qperf results, both the default libnetwork and weave implementation are very comparable, but they are both way off from the host network and flannel test results. I expected them to come really close to the performance of Flannel, because they all use a VXLan based setup.

So why don’t they come close to the Flannel results? All because of the MTU size.

The default libnetwork driver currently doesn’t support larger MTU sizes, but that support is coming in a newer release of Docker.

Weave does support larger MTU sizes, however due to the combination with kernel 4.3 the vxlan implementation of Weave doesn’t work and Weave does a fallback to it’s old userspace implementation.

Changing the Weave MTU size to a high number (8950) on my setup leads to worse performance then sticking with the default MTU size. This problem is very specific to Linux distributions running a very recent kernel, like CoreOS does. Running the same tests on Ubuntu (which still uses an older kernel version) does lead to better performance because then Weave can take advantage of both the VXLan implementation and the large MTU size: https://github.com/weaveworks/weave/issues/1853

Looking at the iperf3 test with 128 parallel connections: the differences are insignificant. This is probably caused by the higher number of packages, created by the high number of parallel connections compared to the qperf test. The increased number of packages becomes the bottleneck, instead of the actual payload of the packet.

What’s next?

Lets hope that Docker 1.10 does include support for larger MTU sizes. And hopefully the kernel problems introduced with kernel 4.3 will be resolved in such a way that the Weave implementation can get optimal performance in jumbo frame environments as well. I also need to include the Project Calico implementation in these results: expect an updated blog on that topic soon. And maybe step away from CoreOS and do a run using some old skool distribution like Ubuntu? Not sure I’m ready for that :-)