Where is the Data r(E)volution headed to?

Unlike in the past, when data was primarily generated in enterprise systems, today data comes from many additional sources: social networks, blogs, chat rooms, product review sites, communities, Web pages, email, documents, images, videos, music and sensors. It is often chaotic - unstructured. The challenge of the data r(E)volution is to unite and process the data, connect the dots, and glean new insights. And, do all this rapidly.

The Data r(E)volution is both a revolution and an evolution.

The revolution lies in:

- New kinds of data, both people-generated and machine generated (e.g., consumer data, query data, sensor data, actuator data)

- Massive amounts of data

- Complexity of data

- Diversity of data

- New ways to organize and manage the data for rapid processing

- New tools for gleaning insights from the data

- New linkages

- Data opening up for more widespread analysis, use and experimentation

The evolution lies in the steady advances of the technology base itself - compute and storage platforms, applications, architectures, and the communication networks that tie everything together.

Data r(E)volution

Analysts predict that the digital universe will be 44 times bigger in 2020 than it was in 2009, totaling a staggering 35 zettabytes. In 2010 Gartner reported that enterprise data growth will be 650 per cent over the next five years, and that 80 per cent of that will be unstructured.

Amidst all this diverse data, connections or ‘link' is surely the king. Linking enables us to connect virtually any set of information together to tell a story; further, the same story can be told in many different ways. To top it off, we're seeing that the connections keep changing. Data r(E)volution is about leveraging data in new ways.

It impacts all industries. Organizations have great expectations to do more with more data. At a business level, the Data r(E)volution is igniting collaboration, innovation, new business processes and speed. At a data processing level, it is demanding new technologies and techniques.

Leading organizations living the Data r(E)volution are setting new expectations for getting results from data. The intent is to create data sets that can be used by a variety of disciplines. In the future, imagine connecting climate data to health data to improve our understanding of climate effects on people. Globally, enterprises are driving infrastructure transformation for (roads, bridges, water system etc.) to reduce carbon emission. An intelligent infrastructure, laden with sensors can adapt so that buildings use less energy or the infrastructure self-corrects for climate-induced change.

The healthcare industry is experiencing a sea change in data proliferation. Analyzing large data sets will be increasingly important for population health (e.g., disease surveillance, provider comparisons, post-marketing drug surveillance) - The vision is to build a network of networks connecting patients, providers, researchers, insurers and health agencies.

Data has been, and will continue to be, the main ingredient in financial services. However, today the data goes deeper - there is more of it, from more sources - and demands more analysis to deliver new levels of competitiveness.

Marketing is becoming more precise and more comprehensive as analytics take hold. Companies are focusing on message evaluation, advertising effectiveness and creative development using powerful analytics to dissect data rapidly and discern detailed behaviour patterns. Social networks are a potential gold mine for marketing, and businesses are challenged to put that data to use.

Thus continues the march of the Data r(E)volution, from building the data foundation to driving new business opportunities. This shift in how we handle data reflects the fact that our world is complex and connected in ways we cannot imagine. The Internet has brought all of this complexity to us in a torrent of data that is at once massive, diverse, disorganized and rich in valuable information. It happened in the blink of an eye, it is here to stay. The data feast is on, and not likely to stop any time soon.

(Paul Gustafson is director of Office of Innovation, Technology Program at CSC. The views expressed in this article are that of the author and do not necessarily reflect the views or policies of CIOL)