Consider this.  This year, you could get an IIM call without attempting any question of English or RC. You could get a call with zero attempts in CAT 2013 from some IIMs. Sounds incredible? The details are even worse if you go by what Prometric is doling out to students in the guise of information and clarifications. Ajab CAT ki Gajab kahaani continues….. Please read on!

CAT has been a computer-based test since 2009. The amount of chaos in the management of that year’s CAT remains legendary. Those of you who are feeling nostalgic or would like to update yourself on the chaos of CAT’09 and the travails of the students that year can check out my video Ajab CAT ki Gajab Kahani!!!

If the first Prometric CAT was so toxic, how could the last Prometric CAT be any different? CAT’13 has again raised a lot of hackles and many questions and accusations are flying around while the IIMs have, as usual, continued with their ostrich-like approach. Let me present here my analysis of CAT’13 and the issues that it raises.

CAT’13: Anatomy of this beast
The highlights of CAT’13 are as follows:

Zero attempt fetches 55.46 percentile
I took CAT on 16th Oct’13 in the 10 am slot and did not attempt any question. But, to my surprise, I have done better than over half the candidates. My performance in CAT’13 is summarized below:

QADI              Attempts 0      Scaled Score 82          53.61 percentile
VALR              Attempts 0      Scaled Score 83          57.23 percentile
Total              Attempts 0      Scaled Score 165        55.46 percentile

This implies that an incredibly large population has obtained negative scores. To be precise, in QADI 93,140 candidates, in VALR 99,430 and in Overall 96,355 candidates have a raw score that is less than ZERO.

As I have confirmed earlier on Facebook, the CAT scorecard doing the rounds with the above scores is mine.

Increased level of difficulty
The various tests in CAT’13 generally have been tougher than the previous year’s. Scaled scores for different percentiles in CAT’13 have been noticeably lower than last year’s scores at the corresponding percentiles.

As an illustration, in previous years, 100 percentilers in each section of CAT used to have scaled score of around 220 in the section. This year, scaled score of around 205 has been good enough to get 100 percentile in QADI and a scaled score of around 190 has been good enough to get 100 percentile in VALR.

A consequence of the increased difficulty is that to get a score of 97.5 percentile one needed to get 15 net correct answers in QADI (scaled score of 137) and only 9 net correct answers in VALR (scaled score of 135). Thus a student scoring 97.5 percentile in both the sections would have an overall scaled score of 272 and 98.80 percentile.

These highlights of CAT’13 have thrown up lots of very uncomfortable questions that the IIMs and Prometric need to answer.

Zero attempts can lead to final selection in an IIM – Can you imagine ?
In IIM Kozhikode’s PGP 2012-14 batch, the minimum CAT percentile at which a student secured a seat was 55.29 in the ST category. Similarly, IIM Ahmedabad had called candidates with percentiles as low as 47.74 (ST category) and 46.22 (DA category) for the personality assessment stage for selection to its PGP 2012-14 batch.

Interestingly, the new IIMs have released their cutoff scores for CAT’13 and have shortlisted students with negative raw scores for the personality assessment stage. Once all IIMs release their data, I am sure we will find that the other IIMs have also done the same – shortlisted (and possibly admitted) students with negative raw scores.

Category CAT’13 Percentile Cutoffs for the new IIMs
Section I Section II Total
ST 41.42 40.80 50.71
DAP 41.42 42.21 50.71

Is a negative score possible for 55% of the candidates?
In the past, a zero score in CAT would have netted a score of around 20-25 percentile. What then led to this significant jump?

The IIMs/Prometric claim that they are not surprised by my score of 55.46 percentile for zero attempts and that such a situation is very much possible. This is a dubious argument and a stupid excuse. As far as possibilities are concerned, a score of 100%ile is also possible at ZERO attempts!!! This will happen when each and every person appearing for the test scores either zero or sub-zero – possible yes, however, very improbable.

To my mind, four possibilities exist for 55.46%ile at zero attempts:

  1. higher difficulty level of the paper
  2. increased recklessness/guesswork by candidates
  3. problems with normalization
  4. software issues

Let us look at these in detail.

Higher difficulty level of CAT’13: Prometric has claimed that it was asked by the IIMs to make a tougher test this time around. This claim is questionable, prima facie, since it was presented only after thousands of students flooded various online forums claiming that their scores and percentiles (particularly in Section II) were entirely implausible. So how do we verify who is right, the students or IIMs/Prometric? The simplest way to solve this puzzle of  “zero attempts for 55 percentile” was for the IIMs and Prometric to share the corresponding data of last few CATs. The CAT 2003 to 2008 papers were among the most difficult CAT papers and data from these papers could also be compared.

However, CAT and Prometric have, as usual, decided to rely on statements and have refused to share any meaningful data. Fortunately, XAT’14 results have been declared and we can use its data to validate the hypothesis that “a difficult paper will lead to higher percentile score at zero attempts”.

XAT ’14 was significantly tougher than CAT’13 tests, it was also taken by approximately 1,00,000 candidates and its candidate profile is similar to that of CAT’13. If what IIMs or Prometric are saying is correct, then XAT’14 should give a higher percentile score than CAT’13 for zero attempts. The comparable data of XAT’14 and CAT’13 is given below.

Section Percentile score for Zero attempts
CAT’13 XAT’14
Quantitative Aptitude 53.61 16.24
Verbal Ability 57.23 3.59
Decision Making NA 4.46
Total 55.46 10-12 (my estimate)

Despite XAT’14 being a more difficult paper, no such bizarre scores have been observed and this comparison clearly and indisputably refutes Prometric’s claim. Actually, it implies that in a more difficult paper students will be more cautious and this will lead to fewer students getting negative scores.

Further, logic also indicates that when a paper is tougher, net scores will indeed go down because the number of attempts by test-takers will reduce, thus, reducing their scores in general. But, negative scores indicate that candidates attempted greater number of questions incorrectly, which is not the case in a test where questions themselves are not erroneous. Thus, higher difficulty level of the paper cannot explain the bizarre result thrown up by CAT’13.

Increased guesswork by students: If students mark answers randomly or resort to unnecessary guesswork then they will get negative marks. But candidates will not suddenly become more reckless than in past years or only in CAT but not in XAT. That a large number of candidates have suddenly become reckless and resorted to random guesswork only in CAT’13, beats logic. In fact, many of these students with negative marks had good performance in CAT’12.

Let me take an example to substantiate this point further. Many students with over 95 percentile in QADI have reported negative scores in VALR. It is reasonable to assume that a student who secures over 95 percentile in QADI will get at least 6 out of the 9 LR questions right, this will give him/her 18 marks and to get a zero (or 55.46 percentile) he/she will need to attempt another 18 questions incorrectly. Or 24 attempts with 18 wrong will give you 55.46 percentile. Many of these students have 20 to 30 percentiles. So, as per Prometric, they have possibly marked 24 out of 30 answers incorrectly. IIMs/Prometric want us to believe that the students who were measured in their approach in CAT last year and in all other papers this year lost their sense of balance on the day of their CAT this year. Incredible!

Software glitches and/or the black box of Normalization: The above implies that the most likely cause of the bizarre result in CAT’13 was either problems with normalization or software issues. The IIMs/Prometric have not revealed their normalization process till date. In the absence of any information regarding this, it cannot be assumed to be perfect. If there is no error in the normalization process, then information about it should ideally be made public. That will put paid to any doubts about this process. Even if they do not reveal the process of normalization, the least they can do is to give the total number of correct and incorrect responses in each section, as is done by SAT.

Further, there is also the possibility that glitches in the software used by Prometric may have led to incorrect capture of responses given by candidates. Last year, it was claimed that marks of 80 students were artificially inflated. If such a thing is possible, then it is also possible that this year the reverse might have happened. The IIMs/Prometric should check their software and ensure that it is free of glitches.

Other Issues with CAT
Bina angrezi kiye saare IIMs ke calls: CAT’13 has the dubious distinction of possibly being the first major MBA entrance test that has made the Language skills (or the VA in VALR) redundant for admission to top MBA institutes in the world.

As discussed earlier, in CAT’13, many test-takers who attempted only 9 questions (all of LR) in VALR, scored 97.52 percentile. This means that candidates who did well in section 1 and attempted just 9 questions in section 2 (all of LR) would be eligible for calls from all the 13 IIMs.

The net impact of this is that it becomes even more difficult for students (including girls) from non-Engineering, non-Science/Math streams to score well in CAT.

IIMs never tire in proclaiming that they want diversity in their batches, they want more girls, more non engineers, more students from non-Math background but every action of theirs seems to suggest just the opposite.

Academic Performance Marks: This is a recurring issue with the selection process of IIMs. As of now, the IIMs, largely, do not follow any normalization process for marks scored by candidates in X, XII and graduation. (IIM Bangalore is the only IIM, so far, which has stated that it normalizes candidates’ marks across boards and academic streams). Thus, marks scored by candidates in different boards are accorded the same weight, without considering differences in the marking patterns/schema of different boards/universities. Similarly, marks scored by candidates in different streams are again given equal weight. In a stream like humanities, it is very difficult to score above 70% marks while in Engineering it is far easier to score in the vicinity of 80% marks. However, the IIMs, largely, do not consider such issues at all. They give equal weight to marks scored by different candidates, irrespective of the candidates’ boards and academic streams. This again skews the selection ratio heavily in favour of engineers. Candidates (including girls) from streams like humanities suffer unjustifiably. The academic profile marks ensure that many students will never get an IIM call even if they score 100 percentile in CAT.

It appears that IIMs themselves do not have sufficient faith in CAT and hence rely on past academic records to shortlist the candidates.

Since the academic profile marks have a virtual veto on the chances of a section of the candidates to get selected to the IIMs, the IIMs should reverse the process. They should first shortlist students on the basis of their Academic Profile scores and then conduct CAT only for these shortlisted students thereby saving money and time of a large number of students. But I am reasonably sure that the IIMs will not do so because the CAT application fee is a significant source of revenue for them.

Incorrect Questions: In the last 15 years there has not been a single year in which CAT was spotless. It is very rare for the IIMs to even acknowledge erroneous questions in CAT, let alone compensating students with marks. In 2006 and again in 2013, the IIMs admitted that there were a few errors in the test. In fact, for CAT ’13, they have acknowledged only one error (on the 10th day of the test), but one of my colleagues is sure that there was at least one error (in section 1) in the first slot on the last day (November 11, 2013) of the CAT window.

If a question is erroneous, there are two kinds of losses: tangible and intangible. If a candidate loses around 5 minutes on an incorrect question, it is a tangible loss. As regards this tangible loss, the IIMs should assess how much time each candidate spent on an incorrect question and should accordingly award additional marks to candidates. In a computer-based test, it should be easy to measure how much time each candidate spent on each page (and therefore, on each question).

The anxiety (and the frustration), which the process of attempting an incorrect question creates, makes candidates lose their balance (one of the most important aspects of a high pressure test like this), which may impact their performance in the subsequent questions too. This loss is intangible.

No one knows how the students are compensated, if at  all they are, for these two kinds of losses? Not considering a particular question is not sufficient.

Leakage of questions: Every year since CAT became a computer-based test, many students, after taking CAT, have uploaded the live questions on various websites, while the test window was still open. Many students would learn these questions by heart before appearing for the test themselves. There was a good chance of such students encountering 3 to 4 already seen questions in their own slots. In a test like CAT, where even one question can be the difference between success and failure, 3 to 4 such questions mean an absolute havoc. Yet, neither Prometric nor the IIMs have done enough to stop this malpractice.

The Last Word
These then are the major issues that challenge the reputation of CAT as well as of the IIMs themselves. The IIMs are some of the most exclusive institutes in the country. As such, the highest standards are expected from them. It is not sufficient that their selection process is fair and true; their selection process needs to be seen to be fair and true and must be above reproach. The IIMs, therefore, need to be absolutely transparent about their selection process. Instead, they hide behind a wall of opaqueness despite the fact that CAT’13 has thrown highly dubious results and thousands of students are, justifiably, venting their frustration at various forums across the web. Is it just a chance that IIMs have now shifted from Prometric to TCS for CAT’14 testing process?

IIMs need to make the admission process more transparent and they need to be accountable.  They must share information regarding the number of attempts and number of correct/incorrect answers with the candidates. The scaling and normalization process should also be shared. Skies will not fall if IIMs do a revaluation of CAT’13 results. Last year XAT on its own did a revaluation of the results on identifying an error, IIMs could follow suit. Students have made the IIMs and not vice-a-versa. If this continues, good students might drift towards XAT and GMAT which are transparent and student friendly.

In IIM Bangalore, I learned about corporate governance, transparency, fairness and the need to carry all stake holders but it appears that IIMs themselves, like most other government entities, do not believe in these. Students – the most important of the stakeholders – have been taken for granted and have been routinely ignored by the IIMs. As a result, I am worried about the reputation of the IIMs in general and of my own alma mater in particular.

I am hoping against hope that IIMs will rise to the occasion and address the concerns of the students but if the past is any indication the ostrich-like attitude will prevail.