Arjan Schaaf bio photo

Arjan Schaaf

Linux, AWS, Azure, Docker & Java rock my world. But far more important: I'm a husband and father of two. Love to BBQ and I'm famous in my world for my burgers & ribs :-)

Twitter LinkedIn Github

For our usecase: apparently so…

Measuring performance & costs

Performance measurement has been high on our agenda over the last few weeks and months. Continuously developing our Amdatu RTI stack, adding support for different Cloud providers and embracing the fast development pace of the Linux Container Ecosystem: it all puts strain on the age old question: how does my application perform? Measuring performance of software applications in general and in our situation mostly stateless web applications is composed out of many different performance indicators. The performance indicators we currently focus on are:

Other indicators like Disk I/O can be very important as well, it al depends on your kind of application. This blog doesn’t try to give a definitive answer on the performance of the Cloud. It gives you some insights on how we addressed measuring the performance of our usecase(s) and hopefully it can help you to decide which performance indicators are essential for your usecase.

How to measure the computing power of your Cloud VM? Not every core is equal

Cloud providers try to keep the information about how their virtual machines perform a bit vague. They talk in number of cores, number of virtual CPU’s, optimised compute virtual machines with even more powerful CPU’s etc etc.

More and more they try to give you extra insights on what kind of hardware is used to run you’re VM’s on: type of Intel Xeon processor, whether or not the VM has an SSD or provides enhanced networking support. But to compare the different instance families within a cloud provider (Azure: A and D series, Amazon: M1,M3,M4,T2,R3 etc etc etc) is hard. Comparing the VM’s offered by different providers like Amazon and Azure is even harder. Even if the specifications suggest similar CPU’s, the overall power of the VM’s can be quite different.

What we did was using Geekbench as part of our analysis on how these different VM types on both MicroSoft Azure and Amazon Web Services perform.

We compared the following vm types:

  • Amazon m4.large
  • Amazon m4.xlarge
  • Amazon m4.2xlarge
  • Amazon r3.large
  • Amazon r3.xlarge
  • Amazon t2.medium
  • Amazon t2.large
  • Azure A0
  • Azure A1
  • Azure A2
  • Azure A3
  • Azure A4
  • Azure A8
  • Azure D1
  • Azure D2

Why these VM types? The m4 and t2 Amazon VM families offer the most generic computing power offered by Amazon, just like the A series on Azure. The r3 VM family is used for memory intensive applications and as such might be interesting for us, because most of the Docker containers we deploy use quite a lot of memory. The Azure D series became interesting after we did a network performance analysis on Azure (Is the network the limit? Dealing with Weave, CoreOS and Azure) Please note that the Amazon t2 family is a bit odd: it offers burstable performance instead of fixed performance like all other types we tested. The benchmarks we ran had enough cpu credits available to run the benchmarks on maximum burst performance instead of the baseline performance, because we wanted to know the maximum performance available and we expect with our usage to not run out of cpu credits. Your milage may vary: it all depends on the kind of load you run on these systems.

Test setup

We used the Geekbench 3 64 bit benchmark suite on CoreOS Alpha 723.1.0, as CoreOS is our preferred deployment platform for our Docker containers. More information on interpreting the Geekbench results can be found here

We focused on the Geekbench Multi-Core score but just the raw score of the VM is not enough. First the price per hour of the VM is relevant as well. A high performing VM can be very expensive. Another factor that is relevant for our usecase the the available internal memory per VM. The ultimate goal of this exercise: find the VM with the best performance per GB of internal memory at the lowest price possible.

Results

The results below are with links to the Geekbench reports with the complete benchmark results:

VM type CPU cores RAM (GB) Multi-Core Score Price per Hour (EU-West) Price / Multi-Core Score Price / Multi-Core Score / RAM
Amazon m4.large Intel Xeon E5-2676 v3 1 8 3346 $0.139 $0.000041542 $0,000005193
Amazon m4.xlarge Intel Xeon E5-2676 v3 2 16 6103 $0.278 $0.000045551 $0,000002847
Amazon m4.2xlarge Intel Xeon E5-2676 v3 4 32 11912 $0.556 $0.000046676 $0,000001459
Amazon r3.large Intel Xeon E5-2670 v2 1 15 3143 $0.195 $0.000062043 $0,000004136
Amazon r3.xlarge Intel Xeon E5-2670 v2 2 30.5 5958 $0.39 $0.000065458 $0,000002146
Amazon t2.medium Intel Xeon E5-2670 v2 2 4 5134 $0.056 $0.000010908 $0,000002727
Amazon t2.large Intel Xeon E5-2676 v3 2 8 5676 $0.112 $0.000019732 $0,000002467
Azure A0 AMD Opteron 4171 HE 1 0.75 552 $0.018 $0.000032609 $0,000043478
Azure A1 AMD Opteron 4171 HE 1 1.75 770 $0.051 $0.000066234 $0,000037848
Azure A2 AMD Opteron 4171 HE 2 3.5 1480 $0.102 $0.000068919 $0,000019691
Azure A3 AMD Opteron 4171 HE 4 7 2716 $0.204 $0.000075110 $0,000010730
Azure A4 AMD Opteron 4171 HE 8 14 5248 $0.408 $0.000077744 $0,000005553
Azure A8 Intel Xeon E5-2670 8 56 16447 $1.97 $0.000119779 $0,000002139
Azure D1 Intel Xeon E5-2660 1 3.5 1896 $0.115 $0.000060654 $0,000017330
Azure D2 Intel Xeon E5-2660 2 7 3695 $0.23 $0.000062246 $0,000008892

Conclusion

Lots of numbers…. lets drill down to the relevant part of all this data. I really like the Amazon t2 family, we’ve had good experiences with the instances over the last year. Especially the new t2.large instance really suits my expectations. Around 8GB of memory is for our usecase the ideal size. Large enough the host a good set of containers, small enough to dynamically scale up & down (larger VM’s with huge amounts of memory would make it harder to scale down and more expensive if you need a tiny bit of extra computing power / available memory). So I’ve chosen the t2.large instance as my baseline. Using that baseline the other VM’s that are relevant to compare to are: Amazon m4.large and r3.large and Azure A3 and D2. Comparing with the Azure A8 instance would reduce the Price / Multi-Core Score / RAM number of hosting with Azure, but the A8 is too big a unit for flexible scalability in my usecase.

Lets compare these instances against the t2.large baseline:

VM type Multi-Core Score Price / Multi-Core Score Price / Multi-Core Score / RAM
Amazon m4.large - 41% 2.1x 2.1x
Amazon r3.large - 45% 3.1x 1.7x
Azure A3 - 52% 3.8x 4.4x
Azure D2 - 35% 3.2x 3.6x

Before I started with this analysis I expected the r3 family to be a serious contender with the t2.large instances. And it is if you just look at the price per available GB of memory. But taking the computer power into consideration as well, the t2.large instance is the best option for us that Amazon currently offers. And Azure…. what’s there to say: disappointing performance at a higher cost with less available memory. We like the IAAS functionality Azure almost as much as we like the AWS IAAS, but facts don’t lie: Azure has some catching up to do before it can compete with Amazon… The relevant VM’s Azure offers are 3.6 - 4.4 times as expensive as the t2.large Amazon instances.