Big Data era is here. Are You Ready?

BANGALORE, INDIA: The world officially entered the age of the zettabyte (ZB) in 2010. That's a 1 followed by 22 zeroes! According to the IDC, by 2020 all of the world's digital data will amount to 35 ZB, 44 times that of 2009. Undoubtedly, the era of Big Data is here.

Big Data is changing how we conduct business, how we live, and even how governments run. In the Big Data era, whoever finds a solution to manage Big Data and explore its value can seize lucrative opportunities near at hand.

Goober, head solutions, Huawei Enterprise India, Big Data is described in terms of the five Vs: volume, variety, velocity, veracity and value.

"We see many enterprises seeking transformation and investing in IT infrastructure construction in this Big Data era. Find a solution to manage Big Data, and you will ensure yourself a bigger stake in the future,'' said he. He describes some of the challenges and how to deal with them.

To manage Big Data and explore its value, two technological issues need to be solved: data storage and data analysis. Big Data, presents the following challenges to traditional data processing and management technologies:

Traditional analysis techniques: Management and analysis of Big Data cannot be realized by traditional IT servers, magnetic tapes, or by vertical expansion (usually by adding hard disks). As data becomes more complicated and exponentially more plentiful, enterprises have to innovate with regard to their information technologies and services. Traditional SPSS statistics analysis software is no longer an ideal modelling tool.

Budget Allocation: At present, a large amount of enterprises' budgets is used for basic business operations, leaving a small portion for supporting commercial operations. It is a concern for chief information officers (CIOs) that more IT facilities are needed, whereas those facilities bring no real added value to enterprise operations.

Automated analysis: According to IDC, more than 80% of Big Data is semi-structured or unstructured data. Many enterprises leave unstructured data processing to a service like Hadoop. However, some investors do not consider Hadoop the best option for analytical query for various reasons, including low performance in certain data mode analysis tasks, high cost in development and maintenance and isolated enterprise information.

In the Big Data era, we see the following situations:

1. Expensive structured data solutions provided by database vendors cannot meet the needs for massive data analysis with tight IT budgets. So enterprises turn to the Hadoop open source system, which eventually results in problems.

2. IT vendors have recognized this problem and integrated Hadoop into Big Data solutions to guarantee Hadoop's position in the Big Data field.

3. Integrating Hadoop into Big Data solutions puts enterprises in a dilemma because their business, archiving, and data analysis systems are all isolated. Complicating the matter, the high total cost of ownership (TCO) restricts the IT budget to a proportion too small for it to thrive.

In the convergent Big Data solution, massive structured and unstructured data are converged. The unified parallel task scheduling framework supports structured and unstructured massive data cleaning and analysis. Through metadata management, a search engine for massive unstructured data is built to accelerate the quick access of hotspot data. And massive data lifecycle management supports archiving and compressing of historical data, and protects data if power failure happens. Convergent solutions provide answers to diversified data storage, data analysis, and "information islands". This solution also meets the enterprise needs for larger quantity and higher performance data processing. Additionally, the convergent solution can be adapted to meet different capacity and performance needs, keeping the TCO at a reasonable level.