Technology fails India’s future managers

For 33 years, the Common Admission Test (CAT) to India's top publicly-run, iconic Indian Institutes of Management (IIM) ran smoothly with thousands taking the conventional paper-based exam every winter.

Occasional leaks of question papers in some centers in a few years or wrong questions being asked were the only blemishes in an otherwise exercise considered to be one of the toughest competitive examinations in the world. For very few countries in the world have 241,000 candidates vying for a few thousand seats in a dozen management institutes.

However, the IIMs who have been running a turf battle with government over autonomy decided to change the format and go fully online for the 2009 edition of CAT. And all the hell has broken lose with the total mismanagement of the examination, outsourced at a cost of more than $42 million (Rs 200 crore) to US-based Prometric Inc.

For reasons unknown, the test was to be conducted over a 10-day window and the callous mismanagement of the online examination in the first 3 days have left nearly 10 per cent of the 75,000 candidates who were called for these days out of the loop. These 7,500 candidates simply could not login to their examinations and now their future career hangs in balance.

Alarmed by the media reports that reported minutely the failures of the online tests from across the country, the four key Prometric Inc people assembled in Ahmedabad, the control center of the exam on November 30.

But the statements of Charles Karnann, COO, Prometric Inc; Ramesh Nova, VP & GM of Prometric in Asia-Pacific; Ramesh Nava, MD of Prometric India and Vijay Thadani, CEO of NIIT, Prometric's India partner for the CAT exam, have left more unanswered questions than answers about the mismanagement of this gigantic exercise.

Prometric officials said some mock tests were done prior to November 28 when the public exams started. Most Information Technology experts in this line I spoke to scoffed at this statement. "If a mock test had been done, many of the faults which hampered the conduct of the exam on Day 1 would not have happened," said an online test expert who requested anonymity.

In fact, IIM's managers too seem to agree. IIM-A director, Samir Barua, told the media, "Given the magnitude of the test, there is no guarantee that it will go right."

All IT experts agree that failures of some computers are always expected. But here the problem seems to be more with the online software developed for the CAT examination itself.

As one expert said, the most fundamental aspect of such a gigantic exercise, that is to "sanitize" the 105 centers where these tests are being conducted, does not appear to have been carried out. All computer systems used in every test center should have been of same quality and a standard anti-virus program should have been installed in every computer and the backups used for the 10-day exercise, said an expert. "How can viruses(computer malware) bring down systems secured for this purpose," asked another software manager.

From information gathered from various experts and statements of students, there appear to be various glaring errors in the process. I list a few here:

For most offline test, candidates are required to report to the test center usually 15-30 minutes before the start. For the first online CAT, candidates were asked to report 120 minutes before the test. Never has any competitive examination made such a requirement so far in the country.

Apparently, the 120 minute window was required to conduct the elaborate biometric identification system developed to prevent impersonation by candidates. " The test's mangers seems to have spent more time on activities peripheral to the actual test," quipped a veteran online test organizer.

At many of the 105 test centers across the country, the computer server crashed when all students logged in. "The basic load balancing test does not appear to have been done before the examination," pointed out another IT expert. The questions were loaded onto regional computer servers and hence the load ( number of students who will be accessing it daily) would be easily known. At every test center, in a batch, a maximum of only 50 candidates were designed to take the test. Unlike other online exams, candidates did not have the facility to take it at any time convenient to them. So the organizers could have easily calculated the load during the examination hours.

Thousands of students who took the exam in the first two days could not access their systems as their passwords were not recognized by the online software. So at many centers, it was learnt, the local officials, gave a common password to all the 50 candidates in each center which allowed them to take the test. " This is a major cause for worry. With a common password there is every possibility of the scores of all those who took the tests in such centers getting mixed up and there is no reliable way to identify each candidates. The results could be unfair to many candidates," revealed another IT expert.

There was a 15-30 second interval between the loading of each question. During this interval, the computer screen were blank at many test centers. Ä software expert pointed out that this indicates the answers were being uploaded to the regional server during this "blank" period and there could be errors in the scoring if this had not happened accurately.

Other online companies which conduct similar tests for top universities and colleges in the country provide multiple levels of back up systems. For example, at every center, there should be back up systems for each one online, two levels of server back with automatic switch to the 1st server if there was any issue with the original server. This was not done at most CAT centers. That is why entire centers were shut down and candidates asked to await future online test dates, due to lack of back up systems.

At most of the centers, candidates were given oral instructions on certain keys on the keyboards which should not be used by them during the test. If a candidate had forgotten these instructions, it is possible that his/her test would become problematic during the evaluation time. "Why where these redundant or irrelevant keys not inactivated automatically using appropriate software, "asked an incredulous online test manager. "This is one of the most essential thing done in online tests."

There are several more questions that come out during discussions with experts. Another interesting aspect is that Prometric, the company chosen to conduct this test through an open bidding process, apparently did not have the expertise to handle the complex test in Indian conditions. Prometric conducts most of its online tests ( GRE, TOEFL etc) throughout the year.

An online expert said at no time would more than 8 to 10 candidates be taking Prometric's test. Compared to this, the Indian CAT is a mammoth exercise with 241,000 candidates taking the test in 10 days at 105 centers. Prometric was chosen from a field of 32 applications which was narrowed down to 4 for the final evaluation during the selection process in 2008.

The claims of some Indian companies who have been conducting online exams for the last 6 to 7 years, accommodating numbers similar to that of CAT every year, were overlooked only due to the American company's "reputation" as an "ace online testing company."

More than anything else, the CAT fiasco has made things difficult for other organizations such as IITs, medical colleges, public sector banks, service organization like railways who too conduct national competitive exams with 200,000 candidates to rethink their decision.

Certainly, the "online test platform" has got a really bad name in the short run due to the callous mismanagement of the new initiatives by the country's top management educators. "If IIMs can't pull it off, who else can," will be the question that will haunt the managers who have been looking technological solutions to handle the enormous demand for elite educational courses in the country.

The last word has not been heard on the first online version of CAT. The cat and mouse game will continue for some more time at the cost of thousands of innocent students who face extra trauma due to the "online uncertainty". If this IIM-induced trauma drives many a potential manager away from this vocation in the coming years, the country will certainly loose the talent.

And all due to the carelessness of the administrators who have so far failed miserably in their immediate task. It would be easier to term this fiasco as a deliberate exercise to test the mettle of India's future managers and their ability to weather adverse situation. But not many will buy such an argument at least for now.