|
Listen to this story
|
ChatGPT can now find bugs in your codes. According to a recent study, ChatGPT is able to find and fix about 77.5% of the bugs. A number which is way higher than its conventional deep-learning counterparts like CoCoNut and Codex when evaluated on the standard bug-fixing benchmark set, QuixBugs.
QuixBugs is rooted in the Quixey Challenge, where contestants were tasked with fixing buggy algorithms in just 1 minute for a $100 reward. Quixey, the mobile app search company held this challenge from 2011-2013. These bugs, developed for human challenges, bring unique interest and reduced experimenter bias, making QuixBugs an excellent dataset for program repair research.
As per the study, ChatGPT’s bug-fixing performance was better than the results reported for the standard program repair approaches. The model was able to fix 31 out of 40 provided bugs. The researchers found that sometimes, the model was able to quickly come up with a solution, while other times it needed more back-and-forth to understand the issue. Despite this variability in bug-fixing, the researchers noted that this could actually be a benefit for end-users, as repeating the request multiple times could lead to a clearer resolution.
To illustrate this point, the researchers gave ChatGPT a question about optimizing a program. They expected it to suggest replacing ‘n^=n-1’ with ‘n&=n-1’, but its first response was, “I’m not sure if there’s a bug without more information on what’s expected”. However, after a bit more back-and-forth, ChatGPT identified and solved the problem in its third response.
But again, is it really helping the developers out there? Or is it just another bug tester that comes with the text editors? The developer community already has compilers that generate warnings about various types of bugs a developer can have.
To understand further, let’s see what ChatGPT can do. For instance, a Medium writer mentioned how he always forgets to import a certain module in his code but still references the module’s function. When asked by ChatGPT, the model was able to identify the problem and fix it.
Tapswi Godara, a developer at Quinnox, weighed in on using ChatGPT as a dedicated bug tester, stating that while the AI model can assist in identifying bugs, the company may not be comfortable with the potential for sensitive IP codes to be made public in a database. Despite its usefulness in individual projects, Godara believes relying solely on ChatGPT for bug testing may not align with the company’s priorities.
“If platforms like Sentry can find a way to integrate ChatGPT without the data slipping to a public database, then it can be a viable solution,” says Godara.
However, Krishna Rastogi, CTO, MachineHack, believes that ChatGPT can still be used in many other ways, if not as a standard tool for bug testing. “For instance, if I have to search ‘regex for letter search in MongoDB’, I’d have to look up on StackOverflow and check out each and every option with most number of upvotes if they work.”
With ChatGPT, Rastogi believes that work has become much easier. ChatGPT not only provides a solution to the query, but also suggests some examples on how to execute the particular code.
Can we really replace an SQL analyst with recursive GPT prompts?
Ken Van Haren, a developer at Patterns, recently wrote a blog where he discussed if it’s possible to replace an SQL analyst with recursive GPT prompts. According to him, SQL analysts dread the work of on-call analytics, where most of the time, the work is to field ad-hoc questions from the company’s various product and operations teams.
To check if the same work can be performed by an analytics bot powered by GPT-3, the developer built CrunchBot, which was basically a free-form question-answering analytics bot on top of the Crunchbase data set of investors, companies, and fundraising rounds.
Needless to say, he was successful in building it, and the bot was able to get correct answers to easy and medium-level questions but got confused on complex questions. For instance, when asked, ‘Who were the largest Bitcoin investors in 2022?’, the bot got results along with the SQL it used.
As per Haren, “playing around with GPT at this level, you get the feeling that ‘recursive GPT’ is very close to AGI”.
However, can SQL analysts really be replaced by recursive GPT? As per Rastogi, it’s not possible. He believes that passion projects like these can help new developers or someone with no experience in coding. “But to say recursive GPT can replace SQL analyst is like saying Grammarly replaced editors,” says Rastogi. He says, languages like SQL are frequently updated, and making a recursive model every time a language is updated is time-consuming.
“So while it can help budding programmers and less experienced ones, one can’t assume that recursive GPT can replace SQL analyst,” concluded Rastogi.
