Harnessing the power of Big Data

Ramendra Mandal, country manager, QlikTech

Today, there is a new universe of data being created by smart meters, mobile devices, social media, RFID, web logs, and other sources. The amount of data is growing as companies gather more and more information with each transaction and interaction with their customers.

It is no longer the case that all possible insights about an organization come only from a structured data warehouse full of vetted data developed inside one's own four walls. Today, there is an incredible amount of interest in ‘Big Data'.

For many organizations its use is an operational reality, providing unprecedented ability to store and analyze incredible volumes of disparate data that are critical to the organization's competitive success, enabling people to identify new opportunities and solve problems they have not been able to solve before.

For many others, Big Data is a big trend in present-day IT that needs understanding and its relevance needs to be separated from the hype surrounding the topic. Embracing big data means accepting that you can gain valuable insights about your organization, your customers, and the world at large from external sources, and by looking at data in a new way.

The present day demands better management decisions, based on more objective data analysis from organizations. This means, better customer segmentation that ultimately helps organizations tailor their products and services. One of the important characteristics of Big Data is that it is often used to store and process unstructured data (e.g., web content such as online reviews, text, social media content) in addition to structured (but highly voluminous) machine data such as data sourced from sensors (e.g., electricity meters), or automated computer systems (e.g., computer logs or algorithmic stock transaction systems).

In order to make practical use of such data, organizations would like to be able to marry this data with existing structured data from their internal OLTP (online transaction processing) systems, data warehouses, and enterprise systems like CRM (customer relationship management) and ERP (enterprise resource planning). By bringing this data together and being able to identify patterns and associations, they can conduct analysis into customer sentiment, customer behavioral patterns, product quality or safety issues, and clinical trial effectiveness.

Why does Big-Data Matter?

Here are two examples of how big data can actually affect our everyday lives. In the US alone there are roughly nine million airplane flights a year, each of which generates data about hundreds of parameters every second or so, from the aircraft and radars and other sources. In addition, each flight has unstructured data associated with it, such as safety updates and reports from pilots and co-pilots. NASA (the National Aeronautics and Space Administration) is using analytics to dig through all this data and gather insights that can identify potential runway incursions and other accidents and prevent them before they occur.

The second example is eCommerce giant eBay, which has a social data intelligence programme in place to help decision makers better understand the company's audiences, influencers, and competitive position - and deliver superior customer service. As of June 2012, eBay had indexed more than 40 million blogs and forums (60 billion posts - 10,000 a second!), which amounts to 65 terabytes of data. A global social analytics team works with multiple groups across the company to find and share insights from all this data.

For commercial entities like eBay, the more data the organization can manage and analyze compared to its competition, the greater its competitive opportunity. For public organizations like NASA, the more data it can process and analyze, the more accurate its predictions can be. These are just a couple of examples of why big data matters.

Right technology for managing Big Data

We are now getting geared up to manage Big Data - technology focusing on providing efficient analytics from extremely large datasets is now available. An exciting development in this area is Data Discovery Platform, which delivers self-service BI. Business users can make in-roads into big data combined with data loaded in memory, without spending time on downloading information. In-memory technology promises impressive benefits in many areas.

The most significant are cost savings, enhanced efficiency, and greater immediate visibility of a sort that can enable improved decision-making. Big Data will always exist. The key is to remember that big data is but a sum of its parts. Broken down into byte-sized bits it can deliver all kind of insight and business discoveries. The important factors are the volume, velocity and variety of the big data - as well as the technology and business brains behind the analysis.

The ‘last mile' with Big Data

One of the big challenges in telecom is the "last mile" - bringing the telephone, cable, or Internet service to its end point in the home. It is relatively expensive proposition for the service provider to fan out the network from the trunk or backbone - to roll out trucks, dig trenches, and install lines. As a result, in some cases they pass high installation costs down to the end customer - or neglect to go the last mile at all. Here is where the problem of "last mile" subsists in Big Data, too.

Today, most vendors working on the problems of Big Data are focused on processing the large amounts of data - they are focused on the backbone, to use the telecom analogy. The last mile: this is where Data Discovery Platform fits into the picture. This is a great complement to the capabilities of vendors focused on processing Big Data and truly provides that high value, highly relevant component of Big Data, namely providing analytics and meaning to their data, for everyone.