There are many recent ground-breaking innovations, including Big Data, Information Retrieval Management (IRM), Social Analytics, In-memory Analytics, and BI Mobility, that go beyond the ability of commonly used BI tools, through more powerful data management systems and capacities of speed and processing optimization.

Each of them empowers the business in accessing the data and insights that they need to make faster and reliable decisions. This article covers about Big Data and IRM complementing conventional BI.

Big Data

Experts acknowledge the recent needs of an enterprise to deal with massive volume, diversity, complexity, and kinds of data that are exponentially generated and consumed in its business systems. This includes unstructured, structured, people- and machine-generated data encompassing social network, Internet texts, emails, images, documents, videos, call detail records, military surveillance, medical records, large-scale e-commerce data, and log files of IT systems.

It is mandated to look beyond the traditional warehouse technologies to handle them and switch to new age solutions for handling Big Data. This should include new people roles (eg, data scientists) and new massive processing abilities (eg, MPP databases, data mining grids, distributed data, cloud computing platforms, and scalable storage systems).

They would now be exponentially increasing the ability to load, monitor, backup, and optimize the use of the large data repositories and its integration across heterogenous data resources.

Schemes would be modeled on a flexible, non-relational, and parallel-relational model. Software frameworks like Open Source Hadoop, Googles MapReduce, Microsofts Dryad, or Yahoos S4 provide the platforms with schema-less design and processing frameworks to provide a sub-second response to queries. Traditional players, including Informatica and IBM Infosphere, work towards exploiting these frameworks.

Information retrieval to complement BI

The blending of search-based information discovery to go along with conventional BI system enables users asking the better questions and reaching the final discovery through continuous refining of questions and tool provided clues. IRM easily adds the capabilities to consume and analyze non-traditional data sources (unstructured data sources) through semantics and text analytics. IRM tools along with BI reports enable self-service discovery by business users.

With predefined/ad hoc search feature, users can do sophisticated searches and keep adjusting their queries to narrow down on exactly what they are looking for. Whether the IRM engine is drawing information from a structured GL system or from an unstructured social media is transparent to the user, who simply sees the integrated results.

The IRM engines, such as Autonomy, Endeca, and Exalead, work in opposite to the conventional data-resides-in-a-schema but that all the data goes into the engine, and then a schema emerges from the attribute relationship available in the data. Ease of search and discovery from unstructured data serves as a potential gold mine for marketing. Sentimental and predictive marketing information could be retrieved with a mouse click from the whole gamut of opinions, comments, relationships, work histories, and personal and professional interests available in the social networks.