About Automatic Benchmarking of IaaS Cloud Service Providers for a World of Container Clusters

15 Jun 2015 Nic McPhee / flickr.com, Lizenz: CC BY-SA 2.0


Cloud service selection can be a complex and challenging task for a cloud engineer. Most current approaches try to identify a best cloud service provider by evaluating several relevant criteria like prices, processing, memory, disk, network performance, quality of service and so on. Nevertheless, the decision making problem involves so many variables, that it is hard to model it appropriately. We present an approach that is not about selecting a best cloud service provider. It is about selecting most similar resources provided by different cloud service providers. This fits much better practical needs of cloud service engineers. Especially, if container clusters are involved. EasyCompare, an automated benchmarking tool suite to compare cloud service providers, is able to benchmark and compare virtual machine types of different cloud service providers using an Euclidian distance measure. It turned out, that only 1% of theoretical possible machine pairs have to be considered in practice. These relevant machine types can be identified by systematic benchmark runs in less than three hours. We present some expectable but also astonishing evaluation results of EasyCompare used to evaluate two major and representative public cloud service providers: Amazon Web Services and Google Compute Engine.

You will find some benchmark excerpts comparing Amazon Web Services and Google Compute Engine from the paper below:

Similarities of Provided Instance/Machine Types

Comparing Processing Performance

Comparing Memory Transfer

Comparing Disk I/O Transfer

Comparing Network Transfer

All measured Instance Types