'Insight' mev Jayate:An episode on Analytics

MUMBAI, INDIA: IF you are one from a tough mountain biker gang, the kind who seeks all its adrenaline from navigating treacherous terrains and thriving on the uncertain, the ‘what-to-expect-next’ suspense; then here’s a question for you.

What if someone armed you with an ultra-smart GPS that can update you of every small turn, every sharp dip and every flat-tyre situation kilometers in advance? Something that even told you when and how much would the snow or rain pour; or exactly when your mate can catch a severe cold.

Chances are (barring the upstarts or nouveau rider league) you would definitely say a resounding ‘No’. For the simple reason that biking is all about being prepared for the vague adventure. Fuzzy logic finds many ardent admirers in this cult. Uncertainty is the chrome and thrill is the fuel that keeps these dudes chugging on.

Till some time back, these lines could have been the best excuse for not asking for real-time churning of insights in a business scenario. But deep down, the gap was bigger than a mountain valley.

In the ocean called information, and around the beaches of social networking technologies or mobility, the term analytics was the surfboard that was almost conspicuous by its absence. Not any more.

Unless one is perched smugly on a Harley Davidson, it’s not hard to guess that this tide is attracting many heavy-metal trials like a magnet.

Social Enterprise: Not a paradox now

“In one form or another, businesses have had ‘analytics’ for decades. But the scope has been too narrow and the technologies too crude. In addition, the embrace and utilization of one-time niche social media websites, which have grown into global phenomenon, have been bungled by most companies.”

These lines cited from “Rethinking Analytics for the Social Enterprise,” a recent research report by experts like expert Don Tapscott and Mike Dover, are a good enough indicator of the pull that the new slew of analytics-based technologies are creating on both IT and business swimmers.

However, many attempts to introduce social networking for employee collaboration have failed because they were not aligned with business objectives. Companies begin adopting social software without a clear vision of what is expected and what is necessary to succeed, as we further observe emerging from this report.

The stock is taken very accurately in a post titled ‘Analytics and the Social Enterprise: Believe in It, or Not’ by Thomas Wailgum, a regular ASUG watcher. He distills the impact of these tools in context to America’s SAP Users’ Group.

When SAP launched a sentiment intelligence tool atop HANA, Wailgum also mentioned how social media outlets, blogs, web communities, wikis and “any other web channel with a publicly available API,” as well as emails and “text data” from companies’ internal CRM systems would be the possible inlets to this application. Incidentally, the system, as SAP had detailed, then “applies semantic analysis using text data processing capabilities from SAP Data Services software, giving business benefits,  like allowing customers to visualize and gauge the marketplace’s attitudes toward their products in real-time–and then react quickly to the events.

Closer to home Persistent Systems was busy exploring the same wave and for a very big stage this time.

The trigger could not have come at a better hour. With the worldwide big data market estimated to be around $3.2 billion growing at a CAGR of 39.4 per cent and slated to reach $16.9 billion by 2015, the advanced data analytics segment was a big pie waiting to be sliced favourably on the company’s big plate of experience in the industry so far.

The piece which is expected to grow at over 25 per cent up till 2015 even in India explains to a large extent the spotlight hogged by names like HANA and Exalytics in most conferences for the past several months.

New advances in in-memory computing are generating a lot of noise for sure. In-memory computing is a disruptive force as per SAP. It is something that provides the speed and agility to power analytics at unprecedented performance levels, while remaining cost-effective. As Maneesh Sharma, Head- Database and Technology, SAP India allows a quick peek, it seems a slew of portfolio additions will keep this territory well-shaken in future too. “Release to Customer (RTC) of SAP BusinessObjects Business Intelligence 4.0 Feature Pack 3 (FP3). This is a very exciting release delivering extraordinary innovation within our BI portfolio. The release of FP3 will facilitate and accelerate the adoption of BI 4.0 whether customers own SAP ERP, SAP NetWeaver Business Warehouse, or SAP BusinessObjects BI platform.”

Translate it all into an analytics page and he claims to deliver more Mobile Analytic Options, Richer Insight, Big Data Enabled with Support for SAP HANA and Apache Hadoop, Deeper SAP Applications Integration and Collaborative Decision-Making.

The surge in analytics and an excited market created an idyllic setting for Persistent with the pioneering attempt of deploying big Data as a technology for the immensely popular and equally debated show Satyamev Jayate (SMJ). Capabilities that enabled Advanced Television Audience Measurement (impact) from various media sources (Facebook, YouTube, Twitter, Google Analytics, web widgets, SMS and phone calls etc) created a different story back stage. A script punctuated with many challenges too as this data was in text, video and audio format and very-content rich, owing to its origin from people opinions, stories and feedback.

Tech-TRPs on Satyamev Jayate

Aamir and our team were very clear from the beginning that the digital strategy was an integral part of the show’s format, says Gayatri Yadav, Executive VP and CMO, Star India.

The team was looking for an analytics/insights partner who not only has the technical expertise in Big Data but also dedication to the show. While several multinational companies in the analytics space were considered in the space, off-the-shelf analytics would not have worked, as the Indian social media users respond in a unique way. She elaborates on the choice of a customized solution.

{#PageBreak#}

“For example, use of “Hinglish” instead of English in responses cannot be handled by readymade tools. Expectation from Persistent was pretty stiff. They were not only expected to process huge number of responses (over one million messages per week), but to come up with dashboards, insights, lead stories as well. All this was expected to be done within four days of the episode for all 13 episodes. It was a pretty high bar.”

Persistent decided to build a software platform to filter/tag contents in a systematic manner from automated first step to manual last step, to find the latest top stories. The results were aggregated and then further used for creating visualizations and dashboards. The analysis was done for all the 13 episodes over a thirteen week period.

Viewer engagement analytics is a very novel concept in India until now, most broadcasters and shows rely only on the TRPs and the hearsay about the impact of the show, explains Dr. Mukund Deshpande, Business Head BI & Analytics Solutions, Persistent Systems.

“Satyamev Jayate was a pioneering show not only in content but also in analytics. The concept is to gauge the impact of the show through analysis of the responses that viewers sent out. The source of the responses included social media platforms like Facebook, YouTube and Twitter, along with the Satyamev Jayate website and SMS and IVR responses.”

A hybrid approach worked well in this case. Specially because the respondents would also use non English languages to respond adding to add the multi-structured and non-textual (audio and video) responses.

“In this approach we chose to use technology as well as human intervention to analyse the responses. The next challenge, given the hybrid approach, was to build a system which would allow us to analyse the data seamlessly both through the technology tools as well as human analysis cycle.” Deshpande reflects.

The tool that made it all possible, as he adds was something that the team built. Its own Content Filtering, Ranking and Tagging System (CFRTS) which is a multi-user, multi-media response analysis system. “The analysis of results was pretty thorough. It went well beyond the simplistic levels such as praise or criticism. Before every episode, we got together with Persistent team, to go over taxonomy of tags for analyzing the responses based of the issue handled in the episode. These tags (more than 50), helped us analyze the responses in the context of the topic discussed.” Yadav adds.

Some of the data and analysis was used by Aamir Khan to show the sentiment of the viewers. Talking particularly of the SMS and web votes on the domestic violence poll, they turned out to be surprising.

“It is clear as a society we have a ways to go on this issue.” Yadav quips. However, when the team looked at the comments a different story emerged. Across the web, Facebook, and Twitter, they saw an amazing response from women, who comprised over 50 per cent of the respondents. “Persistent’s work helped us get a deeper understanding of the messages, highlighting important stories of hope and courage. Women also shared their views on the root causes on domestic violence: Number one, male dominance and secondly, tolerance by women —a very self-aware insight.” She illustrates.

{#PageBreak#}

“The system allows for several users to log on to the analytics platform and simultaneously analyse responses which are allocated to the users by this system. The system also has a provision to analyse the analysis speeds and the data pipelines. The data analysed there would once again be analysed in the second level analysis to ensure that the tagging is correct.” Deshpande offers.

Applause or tears?

The results that the show apparently gained hint about the pudding’s proof. The producers of the show would get real time data analytics during the show helping them gauge the public sentiment; and they could also analyse the contours of responses based on geographical distribution, demographics, time and sentiment of the responses received, as per the company. Moreover, the youth impact of the show could be easily captured and mapped against the views of general populace.

The show had two primary goals, Yadav asserts. “One was to encourage participation and the other was personal impact. The analytics platforms made it possible to do both by listening to social media and people stories. We could have easily have been overwhelmed by the torrent of data received. But with some smart planning, we were able to create a meaningful relationship with viewers.”

The tough part about this tangible technology was however the intangibles, as it turns out. “There is more to emotional expression than just the sequence of words — only human beings can understand the deeper hues of emotions. Sometimes same set of words can have different implications. For instance, “Woman without her man is incomplete” vs “Woman, without her - man is incomplete!”  - no NLP ( Natural Language Processing) tool can understand this.” Opines Deshpande.

Fast processing was also key. And equally crucial were factors like human intervention — deeper contextual references and keeping the content confidential till the show is aired. “Implementing newer revisions to text analytics rules takes time to implement and test, with the human approach — half an hour of instructions is enough to change the course. Also, the show topics were closely guarded and couldn’t be disclosed much in advance which meant that the tool based approach had challenges.”

But the bottom line was about perfection as one would naturally expect from this show. Impact, and that too, actual ground impact can be a really vital reckoner for this attempt. A confident Deshpande affirms strongly here.

“Yes. The platform enabled us to cull responses which needed immediate attention — for example, people not able to donate online or when the show did not have subtitles in some languages. In some instances respondents were directed to appropriate help by the Production team based on the contact information shared by the respondents. As insights partners our job was limited to sharing the insights with the production team — actions had to be taken by them.”

{#PageBreak#}

Analytics: What remains to be drilled

Most companies have not addressed the complex people, process and organizational issues of change, Tapscott and Dover’s report highlighted well. As Wailgum also underlined in his ASUG post,  skepticism around social strategies can not be so easily done away with. “For instance, it’s probably not going to be a blueprint for companies still upgrading to ECC 6 or dealing with a move to BI 4.0.” he writes.

It remains to be seen how all these tools and technologies address issues around this domain. One of them is usability and another one is security issues around consumerisation of IT. Tom Scholz, Research Vice President from Gartner remarked in a conversation once that security implications around BYOD can not be shrugged away. “Most organisations have realized that productivity and freedom part outweighs security factors though. Once you understand the applications and star looking at user profiles from a different angle, things can be sorted out.” He suggests how special category of devices and applications around mobile development strategy can take care of a balanced equation between freedom and security.

R ‘Ray’ Wang from Constellation Research drilled deep into the subject in ‘A Software Insider’s Point of View’. He argues that the arrival of engagement platforms does not signify time to throw out the transactional systems, that serve as the very foundations. The engagement layer exposes transactions and allow for deeper interaction and richer sources of information.  However, the transactional systems lack the ability to support engagement. As to what ERP and CRM vendors have to offer, Wang feel that while many of the vendors have the components for engagement, the struggle will be to embed a sense and respond design point into both the interaction layer and process flows. 

Among the nine key components that Constellation has scoped out in this context, one will also spot Listening and sentiment engines, context engines and decision management.  “Listening and sentiment analysis provides insight into the larger unstructured world.  Relevancy comes from context.  Context comes from roles, relationships, location, time, business process, sentiment, and intent.  At some point, context enables prediction when we shift to experiential systems.” He nails it well.

In case of Satyamev Jayate’s journey with analytics, there were quite a few stakeholders working on it at the same time, and this was a challenge to confront. Besides that the IVR and SMS data was coming from one entity, the Satyamev Jayate website was managed by another entity, the FB and Twitter data would come through a different channel, add to it AKP and STAR TV — which meant coordination was of essence. The utter variety of data, volume of responses and velocity with which people were responding was daunting too.

Data was text, audio and video as well, and in as many languages. The volume grew to over million responses over 13 episodes. A good platform can go out of tune without being able to manage these peripheral pressures. And in such scenarios merely supplying information distorts the whole point. That of providing an actionable insight.

Biswadeep M, Research Director at Gartner feels that even though arrivals like HANA sound a bit hyped, it all boils sown to how one applies the tool. Till then it would be a wait and watch game.

“With SAP HANA, organizations gain real-time insight and analysis of their business operations based on large volumes of transactional data coming from any source across the organization. Equally important, SAP HANA provides a foundation for building the next generation of transactional and analytic applications in the areas of planning, forecasting and simulation” Sharma says addressing the future of applications.

Traditional disk-based data warehouses have limits in terms their ability to benefit from major technology trends such as multi-core CPUs, in-memory processing, and columnar storage, Sharma adds. Good quality data is an important factor and data cleansing are crucial to make any good platform great and worth the TCO, as CIOs see it. “Analytics works better with good data only.” Advises Ajay Dhir, Group CIO at Lanco Infratech Limited.

Social networking, mobility, CRM applications or BYOD (Bring Your Own Device) adaptations; the time is ripe for several so-far-discrete slopes to converge on a new map called analytics. Does the leather last- is a question the answer of which depends on how many questions can it woo the tough-hided bikers with.

Can it take care of queries like: “What trees are best to tie ropes to and what trees augur a bad landslide?” If that can happen, on the move, may be even bravado dudes, for once would run out of some excuses.

Either way, Satya shall prevail.