When ChatGPT Attempted UPSC Exam

With nearly 11.52 lakh candidates appearing for the exam last year, UPSC is often considered one of the toughest exams in the world.

ChatGPT seems to have all the answers in the world. Since its launch in November last year, it has passed some of the toughest exams in the world. In the US, ChatGPT cleared the final exam for the Master of Business Administration (MBA) programme at the University of Pennsylvania’s Wharton School. It also scored better than most students in the US medical exam. ChatGPT also managed to clear Google Coding Interview for Level 3 Engineers.

It was emerging as the world’s brightest kid until it was asked to sit for the Indian Civil Services Examination conducted by the Union Public Service Commission (UPSC)- one of the toughest examinations in the world.

Data Engineering Summit 2023 returns to Bangalore for Second Edition

We at Analytics India Magazine have also been experimenting with ChatGPT for a while and were intrigued by its ability to crack tough exams and interviews. In that spirit, we wondered, ‘Can ChatGPT clear UPSC?’

The chatbot sensed the tough ordeal it was supposed to go through even before appearing for the preliminary paper of the UPSC.

UPSC Breaks ChatGPT

We asked ChatGPT all the 100 questions from Question Paper 1 (Set A) from UPSC Prelims 2022, the answers to which are available on the web. To our surprise, only 54 of them were correctly answered by ChatGPT. 

In 2021, the cut off was 87.54 for general category candidates, so going by the data, ‘ChatGPT did not clear USPC’.

The questions ranged from topics such as Geography, Economy, History, Ecology, General Science to current events of national and international importance, social development, and polity.

ChatGPT’s knowledge is limited to September 2021, hence, it won’t be able to answer questions on current events. However, ChatGPT provided wrong answers for topics such as Economy and Geography that are not necessarily time-specific.

It even got a simple question on History wrong.

In some instances, ChatGPT created its own alternatives. As an illustration, in the following queries, the chatbot presented Option ‘e’ as an answer choice, despite it not being a part of the originally provided options.

The Civil Service Examination is one of the toughest examinations in the world. Nearly, 11.52 lakh candidates appeared for UPSC last year alone. However, only around 5% makes it to the mains. Among those 5%, only a handful managed to crack the mains and get placed as an Indian Administrative Officer (IAS). 

Hallucinatory in nature

Here, it is important to note that ChatGPT is hallucinatory in nature. The same has been acknowledged by Sam Altman, chief executive at OpenAI.

“ChatGPT is incredibly limited, but good enough at some things to create a misleading impression of greatness,” he said. 

Altman believes it would be a mistake to be relying on it for anything important right now. “It’s a preview of progress; we have lots of work to do on robustness and truthfulness.”

Even though ChatGPT has managed to crack some top examinations, it failed terribly in an examination designed for sixth graders in Singapore.  

Researcher Christian Terwiesch, who published the white paper titled, ‘Would Chat GPT Get a Wharton MBA?’ also notes that ChatGPT performed exceptionally well in answering fundamental queries within its area of expertise. However, it occasionally made unexpected errors in middle school level calculations.

Hence, it’s imperative that we consider ChatGPT taking different exams, including USPC, merely as fun experiments. 

In the same inquiry, we also discovered that we could prompt ChatGPT to arrive at correct responses by modifying the inquiry. This means that in certain cases, asking the same question differently could lead ChatGPT to give the right responses, and vice versa. But, during the experiment, we only took into account ChatGPT’s first responses.

Moreover, since it failed on Question Paper 1, we did not go ahead with the experiment with Question Paper 2 and further on. With all the question papers and answers available on the web, anyone can redo the experiment or further it and test ChatGPTs reasoning and analytical skills.

The latest book by Dinesh Pundri is out

The Bloodbath is a lace memory handkerchief, a nasty pair of silky underwear, a blood-stained dagger, and, most significantly, a bag packed with horrible happenings. It gleams with enigma, permeating the skin and erecting the hair’s breadth in veneration and enquiry.

Pritam Bordoloi
I have a keen interest in creative writing and artificial intelligence. As a journalist, I deep dive into the world of technology and analyse how it’s restructuring business models and reshaping society.

Download our Mobile App

MachineHack | AI Hackathons, Coding & Learning

Host Hackathons & Recruit Great Data Talent!

AIMResearch Pioneering advanced AI market research

With a decade of experience under our belt, we are transforming how businesses use AI & data-driven insights to succeed.

The Gold Standard for Recognizing Excellence in Data Science and Tech Workplaces

With Best Firm Certification, you can effortlessly delve into the minds of your employees, unveil invaluable perspectives, and gain distinguished acclaim for fostering an exceptional company culture.

AIM Leaders Council

World’s Biggest Community Exclusively For Senior Executives In Data Science And Analytics.

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox