Big data and its impact on storage technology

BANGALORE, INDIA: Thanks to better communication infrastructure, innovative devices, and low connectivity costs, today, consumers and enterprises are producing and consuming trillions of bytes of information. A huge amount of data is being produced by millions of networked sensors embedded in the physical world in devices including mobile phones and automobiles.

This data growth has an impact, not just on business but on storage technologies too. In 2010, an IDC study predicted that the demand of storage will grow 50 times between 2010 and 2015. The traditional way of storing, managing, processing and analyzing data no longer fulfills the need of present day enterprises and consumers. Disruptive trends such as SMAC and system level developments like Blade Technology, Flash Storage and multi-core processing are driving the demand for innovative and efficient storage technologies.

To address the new requirements arriving out of these disruptive forces, we see the following emerging trends in ‘storage' technology:

Object Storage

According to a 2011 IDC report, the size of the digital universe is more than doubling every two years, and is expected to grow to almost eight zettabytes by 2015. Storing, managing and analyzing unstructured data on such a massive scale creates unprecedented challenges for enterprise and government agencies. These challenges include:

• Preventing data loss

• Maintaining an ‘always on' system

• Protecting data from unauthorized access

• Scaling storage capacity continuously while controlling costs

Organizations are now looking for adopting new storage approaches that are designed to support massive amounts of unstructured data.

Object-based storage is suited for storing massive amounts of unstructured data. What is noteworthy is that, object-based storage systems are not organized hierarchically. Instead, an object is identified and located by its unique identifier. This enables the number of objects to grow substantially beyond the limitation of traditional file systems while still maintaining the integrity and consistency of the data. Organization of the information in an object storage system is generally maintained by the application responsible for reading /writing information.

Cloud storage:

It is eminent that to support virtualization, the storage networks need to be virtualized and dynamically modifiable. Storage resources within the rack/blade systems and outside the servers need to be interconnected and managed as a seamless unified storage system by ensuring continuity. The first one is storage- as- a- service offered by leading cloud infrastructure providers.

In this case the public cloud infrastructure becomes the sharing point for data. In-built storage management systems ensure performance, mobility and security of data across cloud networks. The other trend is to build a storage device that is deployed in an enterprise and works as a local cache for recently used data storage and cloud infrastructure is used as a main storage through a global file system implementation. This model works best for enterprises having multiple office locations. This solution serves as a secondary storage device.

Software Defined Storage:

The world is moving to software driven data centers, wherein all the virtualized storage, server, networking and security resources required by an application can be defined by software and provisioned automatically, there is a need to come out with a storage system that conforms to this. The key characteristics of SDS are policy-driven storage provisioning, separation of storage maintenance from storage media elements, virtual volume and software- only storage functions.

Undoubtedly, the storage industry today is standing at cross-roads and will have to take a holistic approach to be able to blend technology innovations with business demand. Also be able to move the storage software functionality to an optimal place within the stack, without compromising on performance, integration into backup, recovery and business continuity, at an affordable price.

(The author is Director - Business Development & Marketing of Alten Calsoft Labs)