Cloud Benchmarking: Need of the hour

HYDERABAD, INDIA: Cloud in some form of the other has become the most popular technology mantra in every CIO’s dictionary.

We have seen cloud as an important milestone in Gartner’s hype curve and also as a disruptive technology. This has led to sprouting of many cloud providers with varied customer base. Few such examples are; ISV, educational institutes, scientific communities and cloud builders.

The producer of cloud services (Iaas and PaaS) and consumer of cloud services should focus on understanding the importance and benefits of cloud benchmarking. Cloud benchmarking would help both the groups in answering to questions like; 'where do I stand wrt my peers?' Or, 'how do I select the best cloud provider as per my needs with best price?'.

Cloud benchmarking is the concept which measures the cloud infrastructure and cloud services in order to provide insights to the stakeholders (cloud providers and cloud users) for right investments.

Cloud benchmarking framework consists of 'what parameters to benchmark', 'what variables to measure', 'type of tools' and 'execution methodology'. Each topic forms the pillar for cloud benchmarking which is worth exploring and understanding.

Cloud Infrastructure Benchmarking

Ist pillar:

The foundation for any type of cloud benchmarking is the cloud infrastructure. Cloud infrastructure consists of virtualized instances which are elastic in nature. Continuous Benchmarking and measuring of these instances wrt CPU, Memory I/O, DISK I/O and network gives insights about cloud behavioral patterns.

These patterns become critical assets to cloud providers and help them to make proactive decision. Each parameter is measured with specific unit of measurement (UOM).  CPU UOM can be compute speed or compile time speed, memory and network UOM could be latency and finally disk I/O UOM could be latency and throughput.

2nd Pillar:

The QOS parameters like consistency, reliability, performance and scalability of instance/instances has to be also measured periodically for robustness.

3rd Pillar & 4th Pillar:

Selection of right set of benchmarking tools to measure the cloud behaviour depends highly on the type of application or services offering provided by the cloud provider. The selected tools have to be configured and executed in automated fashion to increase efficiency and reduce errors.

Cloud Services Benchmarking

Once the cloud infrastructure benchmarking results gets stabilized then we can start the benchmarking exercise on cloud services itself and compare the results with various cloud providers. These services could range from distributed DB service, storage services, caching services etc.

Again depending on the type of services, the right set of benchmarking softwares has to be developed with right set of scenarios. This will help the cloud providers to stay competitive in the business. On the other hand, it gives an opportunity for ISV (Independent software vendors) to shop around and get the best price.

Cloud benchmarking is gradually gaining momentum and we will see cloud benchmarking as one of the main feature in every cloud providers roadmap.

The author is senior program manager with the Developer Tools team at Microsoft India Development Center.