DARK DATA – Goldmine for Industries

In the current age, industries are debating on digital transformation and digitization of systems, and in this whole transforming world, data is our driving force. With the growing interaction between data and people, it is astonishing to see the new business opportunities getting unlocked for companies across the world. Just imagine, when 5G and 6G comes into effect and the speed of data transfer becomes 1 TB per second, it would result in a plethora of data. Hence, it is extremely important for businesses to radically re-align how they operate.

While data will continue to drive every business forward, there is enormous raw & unstructured data getting generated by social media, IoT devices, mobile devices, audio, video, search engines etc. every day. They hold vast, hidden opportunities which can be explored and leveraged for good business insights and strategies.

According to IDC, “90% of the unstructured data are never analyzed. Such data is known as dark data.  According to Gartner, “The information assets organizations collect, process, and store during regular business activities, but generally fail to use for other purposes.”    

Excess of data is being generated by enterprises and industries, so why is all of this being stored? Some of the following focus areas could be leading to dark data accumulations:

Affordable Infrastructures

Cheap data storage – Today, the affordability in storage cost has enabled organizations to store their relevant and easily accessible business data, which could be structured, semi-structured or unstructured.

High processing power to collect information - Technology advancement is such that there are many event processors which could collect any kind of event data and store them. Now, these unstructured data are enormously huge and relevant information goes unnoticed and unexplored.

Regulations and Tools

Governance and regulations strictness - Today, companies have started to accumulate every bit of information and have set multiple regulatory measures which enables them to provide any sort of information to auditors and for later analysis.

Lack of Sophisticated analytical tools - Technology advancement is moving very fast but at the same time, we are still waiting for technologies which could help analysts read the unstructured data and decipher the inference.

Dark Data Analytics 

Until a few years ago, data and analytics were standard with the ultimate goal of collecting, processing, and displaying different sorts of reports. Descriptive Analytics, which later moved to Predictive Analytics and forecast incidents where businesses started gaining. Now, analytics is maturing where industries are not only looking for prediction but Prescriptive Analytics to understand the root cause analysis. With such analytics, all sorts of data (structured, semi-structured, or unstructured) explorations are done, and dark data plays important roles in it. Some of the leading industries which are benefiting in a big way are:

Education Industry:

The education industry is rapidly adopting the digitization. Institutions are shifting towards virtual classes; building next-generation systems and that is where “Adaptive Learning” plays a critical role. Systems are being designed according to the learner's ability and this requires gathering all information pertaining to an individual's pattern of learning, their grasping power by knowing how much time the learner spends on one topic, level of IQ, etc.

Accordingly, suggesting the next course to learners while they are learning, what is the best medium for students to learn the concept (audio/video). So, these are behavioural (unstructured) data captured through web/mobile medium and analyzed for deriving inferences and insights. Hence, leading to a greater level of acceptance by students and ease of learning which indirectly leads to better business and growth of the education industry.

Retail Industry:

Another big industry which can be heavily benefited with such data is Retail industry. This industry is completely driven by consumer behaviour. Though the retail industry is shifting towards digital shopping, there are many opportunities to study consumer behaviour using dark data.

If we talk about retail stores, then we think of CCTV cameras which are generally used for security reasons only. However, this streamed data can be utilized to read consumer behaviour which could bring out different insights about customers.

Customer Service:

Any customer service team where they share only specific data are useful for a particular instance. However, their recorded conversation holds other relevant information about the customer which can be utilized by the sales team to improvise the sales as well as customer satisfaction.

If such information is not collected, then it would be difficult to take out inference and could result in customer churn. Similarly, other industries like hospitality services which are mainly dependent upon customer satisfaction or medical industry where patient behaviour or their medical transcript can offer terrific opportunities.

Sunil Singh, Director – Engineering, GlobalLogic

This clearly states how relevant information is lying undiscovered and the plethora of opportunities offered by dark data can be leveraged profitably by organizations across industries.

How to reap the benefits out of Dark Data?

Industries are aware about the importance of such dark data and to enlighten them we could focus on these specific areas which could help derive important insights.

• Sharp analytics team who can mine right information out of dark data.
• Relevant tools and technologies which could ease the task of the analytics team.
• Better strategic planning by data stewards or domain experts so that unstructured or semi-structured data could be transformed and be utilized in the best possible manner.

Bottom Line

Data will continue to grow with massive information hidden in its appetite. Recent advancement of technologies like deep learning, cognitive computing, behavioural analysis, pattern identification etc. are being used to derive inferences and results. Going forward, analytics needs to focus on strategic planning, processes, tools, and techniques with a sharp team who would be required to jump into Dark data and get the relevant information out.