When ChatGPT was launched, it was said to have all the answers in the world. AIM took the test of that promise, and made ChatGPT attempt the Union Public Service Commission (UPSC) examination. And as we all know, it did not clear USPC. 

Now that Indian language models are all the hype, we wanted to make at least one of them attempt the country’s most prestigious and one of the toughest examinations in the world. Since the challenge was so tough, AIM decided to test out Krutrim, which is touted as the most indigenous and culturally aware model of India.

To make Krutrim realise the tough ordeal that it’s going to go through, we decided to let it know and asked if it thinks it can clear the exam. Unfortunately, instead of accepting its fate, it decided to wish us ‘Good luck!’.

Not So Smart and Aware

We made Krutrim attempt the 100 questions from Question Paper 1 (Set A) from UPSC Prelims 2023. It only got 41 of them correct. Since the cut off of the exam was 75.41 for the general category this year, Krutrim failed the UPSC exam miserably. 

To compare, ChatGPT answered 54 of them correctly when we took the test in 2022.

The questions ranged from subjects such as geography, economy, history, ecology, general science to current events of national and international importance, social development, and polity. 

Strong at Small Questions, Weak at Reasoning

When it comes to geography and general science, Krutrim was able to answer several questions correctly. But, when it came to history and economy, the chatbot fared poorly at even understanding the questions. But all of this seems to depend on its mood. 

Moreover, if the given questions had longer contexts, Krutrim failed to correctly answer almost all of them, showing its weak reasoning skills.

Since Krutrim is not connected to the internet, it was not able to answer any questions on current affairs. Surely, it is still in beta and with future updates, the model will be able to get real time information, and maybe hallucinate less too. 

Its responses were at times difficult to understand.

The Context Window Problem

Another problem that Krutrim faces, which is worse when compared to other AI models, is that users cannot insert all the text from a single question in one go. At most, Krutrim can take an input of about 500 characters, which is roughly 80 words. Many questions from the paper were longer than that, thus Krutrim could not process them hassle-free.

Moreover, although Krutrim claims to support multiple languages, pasting questions from the paper into the input box was impractical because it counted those characters as more than their English equivalents.

Plus, there is no option to upload a PDF or even scan images on Krutrim yet, which could have made things a lot easier. Nonetheless, attempting the paper in Hindi or other Indian languages is for another time. 

Not All is Lost

This just clearly points to the fact that Indian language models, in this case Krutrim, are not nearly as smart as say ChatGPT or Perplexity. Krutrim struggles to find the right answer, and even if it does sometimes, it is hard to assess if it was a fluke or not, since there is no concrete explanation for the answer. 

The attempt was also made on the browser, and not the app released by Ola Krutrim since the text input method is far worse in it, and is also without voice input. 

Though Bhavish Aggarwal, the CEO of Krutrim, is making big strides in the country with a lot of announcements to make Krutrim the best Indian AI model, such as creating its own cloud and offering its API to developers, there is still a lot that needs to be improved.

Meanwhile, when we informed Krutrim that it had failed the UPSC exam, it told us that we had failed the exam and how it can assist us with study materials! 

Well played, Krutrim, well played!