Did BI lead to the Big Data revolution?

BANGALORE, INDIA: We have always lived in data intensive environments, and organizations which rely, strategize on the basis of this data are the organizations that succeed in today's times.

Studies show that we are generating enormous amounts of digital data on a day-to-day basis. In-fact, the growth of data sets have become so large that they defy the ability of conventional forms of information technology to store, manage, analyze and gather important insights from them.

According to research firm IDC, all of the new information generated in the world in the year 2000 amounted to about 2 million terabytes. Now, the digital universe generates more than twice that much in a single day.

The practice of analysing data to gather business insights has been in existence since the late 1950s when technology captains defined business intelligence (BI) and its benefit to the business world back then. As time evolved and IT became an indispensable function for any organization, BI tools also evolved to provide greater insights to the business.

Traditional business intelligence (BI) tools typically crunch numbers to report what happened in the past and why. This prevented the broader organization from taking advantage of all of their data, systems and people - and its ability to become more data-driven and competitive.

Although (BI) tools have been extremely proficient in tracking raw transactional details like sales figures and profit margins, they were not useful in tracking the root causes, or drivers, of trends from that data.

What was missing from this picture was the power of predictive analysis. While BI tools were able to reveal basic facts and sequence of events, they lacked in analyzing deep insights which alerts any business of transformation trend patterns.

This boosted forward looking organizations to take up Big Data solutions and dig deep insights to augment their business requirements. 

Another reason that led organizations to enhance their analysis capabilities was the advent of social and interactive web applications, which led to data explosion. All of this unstructured data proved to be either a goldmine or a colossal challenge for organizations depending on their technological advancements and business outlook.

BI tools are directly linked and as effective as the data that goes into them and their supporting data warehouses. This means BI tools were effective only with structured data and only those organizations, equipped with an effective data retention strategy could derive substantial value out of BI.

However, traditional BI tools proved to be ineffective for mining and analyzing unstructured data which can debatably be determined as a game changer especially for consumer oriented businesses. Mapping those unstructured data sets is another reason that inspired Big Data to overcome this challenge.

{#PageBreak#}

In competitive environments, it became crucial for leading organizations to evolve from basic information about past trends to predictive trends to sustain their leadership position. Modern businesses had to respond to the rapid proliferation of data that surrounded them. It became critical to deal with the speed at which data was created, identify with the volume of data created, and assess the relevance of this data, all on a real-time basis.

Traditional BI tools were not able to address these factors as they worked primarily on structured data and on the premise of feeding data to warehouses or similar tools. Now, with social media becoming more and more dominant, there is a lot of unstructured data being generated.

As such, predictive analytics and the capability to access external and unstructured data sources in real-time is now being possible with Big Data technologies.

Big Data technologies can capture and analyze data from very large data sets (structured or unstructured). For example, Big Data technologies can evaluate data from facebook pages, twitter feeds (at the basic level) and analyze it with an organizations' sales information to deliver critical market intelligence, helping companies understand trends and consumer sentiments, predict churn, respond to customer issues and tweak strategies in a more timely fashion.

Another example is the retail sector. Big retailers look to gain advantage by understanding the behavior and perceptions of their target markets and competition strategies especially during festive seasons. And they would typically need this kind of information on a real-time basis; for example, a supermarket will find immense value in data that shows a competitor's marketing strategy for a festive promo based on the first day of the promo's launch.

Since the need of the hour is to analyze this data on a real-time basis, the retailer would not be able to use BI tools for this. On the other hand, by leveraging Big Data technologies, the retailer can gather data feeds around how consumers are responding to promos put out by competition (provided he gets this data real-time), and create pattern-based insights.

In summary, while BI and traditional data warehousing tools focused on managing and reporting existing business data, Big Data technologies generate predictive insights and deep analysis through advanced analytical tools and algorithms.

With digital data surmounting and the buzz on Big Data rising to significant levels, it is important for organisations to understand the difference between their existing investments in BI tools, and if they need to invest in Big Data technologies.

While applying forward-thinking analytics to Big Data sets has obvious appeal, does it supplant the role of traditional data warehouse and BI efforts or should Big Data analysis and business intelligence be considered as two ends of the same spectrum?

As CIOs look into Big Data, they will realize that while "Business Intelligence" is the underlying concept which led to Big Data technologies, BI tools and Big Data technologies have different roles, capabilities and implementations. Not all organisations currently using BI tools need to have a Big Data strategy, and not all Big Data implementations need to be supported at the back-end by BI tools.

It is hence prudent for vendors to advise customers on the need and implementations of BI tools and Big Data clearly, so that the power and purpose of both are well understood.