PostgreSQL Elephant Dash: Cloud Application Benchmark of 5 IaaS Providers

To see the latest results, please click below.


See More Provider Comparisons Free


View our previous post on general server performance of 5 large IaaS providers.

Disclaimer: we were not paid by any IaaS provider to run these tests, nor do we endorse any of these providers. The results are laid out from an analysis of the data available from PostgreSQL’s native benchmark application installed alongside its database application.

Purpose


Databases are a highly popular workload in public cloud environments. PostgreSQL is a highly scalable, feature-rich database that is a popular solution for businesses. This document explains the 8-day performance results of PostgreSQL’s native benchmark on 5 of the largest cloud providers in the world: Amazon EC2, Google Compute, HP Cloud, Rackspace Cloud, and SoftLayer CloudLayer Compute. An analysis on performance only, as well as price-performance, is detailed below.

While we run synthetic tests to gauge the performance of cloud servers as well as those servers’ CPU, RAM, disk, and internal network performance, nothing tells you the true performance of an application like, well, actually running the application.

Chart 1.1

Results

The Takeaway:

  • If you would like an overview in performance over the 8-day period, see Table 1.1.
  • If you are interested in strictly performance-based results, see Chart 1.2. The average is over a period of 8-days highs and lows.
  • If you are interested in price/performance results, see Chart 1.3. The higher the score, the more value a provider delivers to its customer for every dollar spent.

Table 1.1 below summarizes the results of the 8-day continual benchmark of 5 of the largest IaaS providers in the market. Google Compute, a recent entrant to the market, has been included in this analysis to provide insight into its performance capabilities. While a strong contender on average over the 8-day period, Google Compute does not run the PostgreSQL test as efficiently as SoftLayer’s CloudLayer Compute.
Table 1.1

Result Amazon Google HP Rackspace SoftLayer
Average 713.07 1441.75 1170.1 587.31 2408.87
Standard Deviation 101.19 71.03 39.22 116.18 298.11
Coefficient of Variation 14.19% 4.93% 3.35% 19.78% 12.38%
CloudSpecs Score 32 60 90 26 100

Chart 1.2

Chart 1.3

The Most Stable Elephant: HP Cloud
To determine the Most Stable Elephant, we used the Coefficient of Variation (CoV), which is calculated by dividing the 8-day average by the standard deviation (a measure of the spread around the average used in our case to understand how much performance fluctuates over time). The CoV gives a percentage to depict the relative fluctuation of each provider, rather than a numerical spread. The percentage makes it easier to compare among providers to see who has the most stable performance–the lower the percentage, the more stable the performance. Stability wise, HP Cloud has the lowest CoV over the 8-day period at 3.35%. By contrast, Rackspace Cloud has the highest CoV at 19.78%. Relatively speaking, HP Cloud shows almost 6x more stability over this 8-day experiment.

The Performance Elephant: SoftLayer’s CloudLayer Compute
We use the average score on Table 1.1 to determine the Performance Elephant. Over the 8-day period, SoftLayer’s CloudLayer Compute offering scored highest on average at 2408.87.The lowest performance scorer, Rackspace Cloud, scores around 4x less at 587.31. While it scored higher than any other provider tested, it did not provide satisfactory stability when compared to Google or HP Cloud. It should be noted, though, that even SoftLayer’s lowest-performing PostgreSQL score (1736.9) is still higher than any other provider’s highest score (Google, the second highest average, has a high score of 1584.34).
The Price-Performance Elephant: SoftLayer’s CloudLayer Compute

The CloudSpecs score is used to determine the Price-Performance Elephant. By dividing each provider’s score by its cost per hour, we receive a normalized score. Taking the normalized score of each provider and dividing it by the largest normalized score, we calculate a relative percentage (with the highest provider receiving 100) that can be used to compare the price-performance value among the providers. SoftLayer’s significantly higher score over the 8-day period also makes it the price-performance winner of the experiment (with a CloudSpecs score of 100), despite the fact that it is the highest-priced provider (tied with Google Compute at $0.26 per hour). HP Cloud comes in as a close second with the advantage of being the lowest-priced provider on the list.

Server Setup


Out of the providers tested over the 8-day period, SoftLayer’s CloudLayer Compute offered the most flexible server build, allowing for independent allocation of vCPUs, RAM, and disk. Amazon, Google, HP, and Rackspace all deliver packaged offerings with pre-determined allocations of each respective resource; therefore, it becomes a complex task to match the exact offerings of each provider. In this case, we controlled for CPU allocation.

*Google’s offering, which is actually 4 vCPUs, was throttled to only operate with 2 vCPUs, so it did not receive any performance advantage for the purposes of this study. Its RAM allocation is also slightly below 4GB RAM, at 3.60GB. Amazon’s closest 2 vCPU offering, its m1.large, allocated 7.5GB RAM.

Table 1.2

IaaS Provider Amazon Google HP Rackspace SoftLayer
vCPUs 2 2/4* 2 2 2
RAM (in GB) 7.5 3.6 4 4 4
Disk (in GB) 50 56 120 160 100
Cost per Hour $0.24 $0.26 $0.14 $0.24 $0.26

A Note on the Pricing
We understand that many providers offer discount pricing with longterm commitments; e.g., Amazon offers competitively priced Reserved Instances, where the cost of the virtual machine is inversely correlated with the commitment of time and upfront cost (the longer a user commits, the less he/she has to pay); similarly, other providers offer additional incentives, such as SoftLayer, which offers free bandwidth for a monthly server purchase. With this in mind, a provider’s CloudSpecs Score, which is dependent on the pricing, can fluctuate. While we acknowledge this, for the purpose of a basic understanding of each provider’s 2vCPU offering under a PostgreSQL workload, we opted for the hourly pricing to avoid confusion, as the purpose of this article is to show a variety of strategies to approach vendor selection.

About the PostgreSQL Benchmark


This is the default PostgreSQL benchmark (pgbench), which is also available in the Phoronix Test Suite. We use the Phoronix Test Suite to run this application test within the virtual environment. The tests, including pgbench, are run with default parameters; i.e., nothing was changed.

By default, pgbench runs a transaction simulation. Per transaction, it runs 5 SELECT, UPDATE, and INSERT commands. The actual commands are run in this order:

  1. BEGIN;
  2. UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
  3. SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
  4. UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
  5. UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
  6. INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
  7. END;

Table 1.3

pgbench Information Value
Phoronix Test Suite Version 3.4.0
PostgreSQL pgbench Version 1.4/td>
Original Project Website http://www.postgresql.org
Estimated Run Time 2077 Seconds
Download Size 13.83 MB
Environment Size 230 MB

Please email us if you have any further questions pertaining to this article. To learn more about the benchmarks we run, or to work on a benchmarking project with us, please contact us.

View our previous post on general server performance of 5 large IaaS providers.