NEW DELHI, INDIA: Picture this- You have a gigantic gas-guzzling datacenter housing thousands of servers and computing components. You are experiencing an ‘information log’ and this information is mission critical to your business.

You feel the dire need to store, protect, intelligently manage, virtualize and automate this juggernaut of intensely complex information assets. In order to add efficiency, you deploy additional information Storage infrastructures and you just keep on adding them as the volume of information grows. 

Until one day you find that the data center has clogged out 3/4th of your office space and emptied your company’s exchequer. Your expenses have reached its zenith in lieu of hardware spend, cooling costs, power and other overheads. Still, you have not been able to achieve your end objective- ‘Manage Information Better’.

Welcome to the age of information overload! Today every individual and enterprises are grappling with the burgeoning complexity and quantity of the information created and replicated every day. According to an IDC study “The Expanding Digital Universe: A Forecast of Worldwide Information Growth Through 2010” the amount of digital information created, captured, and replicated in 2006 was 161 billion gigabytes.

At that time, it was expected that in 2007, for the first time ever, the amount of data generated will have exceeded the total storage capacity that was available then. It is projected that the information added to the digital universe in 2010 will be six times its size in 2006.

The incessant growth of replicated data leave many organizations in a quandary in terms of managing and storing data and making them available for backup, disaster recovery, business continuity, governance and compliance reasons, apart from using them for attaining business objectives and competitive edge.

Enterprises are juggling with the growing volume and complexity of information as Backup and recovery through servers, networks and applications has failed to keep pace with the enterprise growth trajectories.

Traditional backup solutions store data repeatedly and incrementally, expanding total managed storage by 5 to 10 times. Also, many organizations operating in the global economy are experiencing exponential data growth in remote offices coupled with the increasing importance of data to run distributed operations. Added to this is the fact that organizations are increasingly finding it necessary to operate 24/7.

The need to backup data from remote offices with the same level of protection as provided at headquarters while having less time to carry out the backup has placed increasing pressure on organizations to find ways to make better use of their available network bandwidth.

Further, in case of traditionally used backup solutions, because of unnecessary data movement, enterprises are often faced with backup windows that roll into production hours, network constraints, and too much storage under management.  Similarly, in the case of virtual infrastructure environments, server consolidation can mean overlapping backup windows and heavy impact on hardware resources like CPU, Memory, NIC cards etc.

Data explosion is a challenge that all organizations are universally facing. Although the price of storage continues to decline, the brute force approach of adding both storage capacity and network bandwidth is obviously unsustainable.

Organizations recognize the urgent need for solutions to contain and mitigate the effect of data explosion without compromising the effectiveness of their backup and storage processes- The answer lies in ‘Data de-duplication’ technology.

Let us discuss how this technology actually operates- Consider the sentence “Companies can benefit from data de-duplication for quick, efficient backup and recovery.” The first time this sentence is backed-up, the de-duplication technology will fragment it into its component words and backup all twelve words.

At a later time the sentence is edited to read, “Organizations can benefit from data de-duplication for fast, efficient backup and recovery.” When the changed sentence is backed-up, the de-duplication technology will again fragment it into its component words and compare these words with the previously stored words. “Organizations” and “fast” are new words and so the de-duplication technology will add it to the backup record.

However, the other ten words already exist in the backup record so the de-duplication technology would only record unique ID pointers for those words. Therefore only one copy of each word in the two sentences is recorded—14 words instead of 24—yet both sentences have been fully backed up.

Similarly, when an email with an attachment is received by multiple recipients within an organization, data de-duplication technology only backs up the first instance of the attachment. All other backups simply point back to the previously stored instance of the file. By reducing the amount of redundant data that is backed-up, data de-duplication technology reduces the data storage capacity and the amount of network bandwidth an organization needs.

Thus, data de-duplication technology facilitates better storage resource utilization. A multitude of organizations still use tape as part of their backup strategies although disk storage is known to be more reliable and offers superior performance. With data de-duplication in use, disk storage can offer the same economies as tape storage.

The use of disk-based storage improves RTOs (recovery time objectives), offers superior reliability and improves the efficiency of WAN-based remote backup and replication. An average organization will be able to store 20TB of backup data on just 1TB of disk space. As a corollary, it would lower power, cooling and accumulated overheads cost to a considerable extent.

As data de-duplication frees up considerable disk capacity, organization can utilize it to bring other backup data presently on tape onto disk, and/or to allow longer retention of data that is backed-up to disk. This means that organizations can recover data that is several months old without having to go to tape. Tape-based backup can then be reserved for long-term archival of data that is infrequently accessed and for “doomsday” data recovery.

Data de-duplication is either performed on the backup server (the source) or on the VTL or disk appliance (target). When deployed at the source, data de-duplication software reduces the amount of backup data that needs to be sent over the network each day.

With less physical data traveling over the network the expense of WAN-based remote replication falls so that some organizations can adopt this type of data protection policy for the first time while others can expand their existing policy to include remote data previously not protected over WAN links of as low as 128 kbps.

The downside is that source-based processing requires that you rip out and replace your existing backup application or live with two different backup applications in your environment — one for your remote-office backups and one for your data center backups.

The advantage of de-duplication at the target is that it’s much less disruptive to your existing backup environment. It’s not necessary to replace your existing backup application, and in the case of a VTL, you don’t even have to alter backup policies and procedures.

Thus, we can conclude by saying that Data de-duplication is a revolutionary technology.  Whether the need is to improve return on investment by reducing storage infrastructure and management for both backup and archive – or to address environmental issues by reducing the data center footprint for backup along with attendant energy and power savings – data de-duplication has a significant role to play.

According to industry estimates, the data De-duplication Market opportunity will touch $1B by year 2009. Data de-duplication technology has the potential to change the economics and value of data management and protection, in today’s scenario.

The author is VP, Enterprise Business EMC India and SAARC.