It seems that every day there’s a new AI tool rushing to take up developers’ jobs. First, it was GitHub Copilot, taking on generating code from developers —— from delegating them to debugging. And now, even debugging is being taken away as researchers have found a way for LLMs to self-debug. 

Earlier this week, a paper published by Google Brain researchers demonstrated a technique that allows LLMs to debug their own code. Using this method, LLMs were able to improve their accuracy by up to 9% on text-to-SQL generation and close to 10% on code translation. This method, touted as an early improvement, along with others, could invoke a paradigm shift in DevOps. 

Making LLMs better coders

The paper, titled Teaching Large Language Models To Self-Debug, describes a novel method that allow for self-debugging. The approach leverages one of LLMs core strengths —— the capability to explain code in natural language. By making the model explain itself in natural language, it is able to identify its own mistakes and correct them.

While this approach may appear extremely simple at face value, evidence suggests that it is effective. It allowed LLMs to achieve state-of-the-art performance on various code-generation benchmarks such as the Spider dataset for SQL, Transcoder for C++ to Python translation, and MBPP for Python. Even beyond creating better code, self-debugging improves sample efficiency, meaning that each output is more ‘valuable’ than an output that doesn’t use self-debugging. 

The researchers stated that this approach was inspired by human programmers, who too rarely get the code right on the first attempt, instead taking a second look at what they have produced. Bringing this capability to LLMs has been tried in the past with other research papers like Self-Refine and Reflexion. However, self-debugging seems to be the approach which uses the least amount of resources and provides the most optimal output. 

The workflow for self-debugging is fairly intuitive. In the first step, the model generates code, which is then executed. While this code is executed, the model explains the code, which is then bundled with the execution result to make up a ‘feedback message’. This message is fed back into the LLM, and the process begins all over again until the programming runs successfully. 

Along with this approach, the researchers combined few-shot prompting and execution-based code selection. Few-shot prompting allowed the LLM to learn the input-output syntax, as well as integrate the explanation generation component with every output. Execution-based code selection allows the LLM to select the best final prediction from a range of samples, thus increasing the sample efficiency.

As seen by the paper, the model iteratively improves itself by giving an explanation for each wrong output. While this method might involve more steps than accepting the code from the LLM directly, it does not require any additional training or fine-tuning, meaning that it could possibly be integrated easily into existing code-generating LLMs. 

With this approach and others, the next generation of auto-coders might just be self-debugging. This brings up a very important question: will it be a disruptor, or will it be an enabler?

10x developers or 100x developers?

While some see autocoding platforms as a threat to their profession, others feel that these have the capability to supercharge the capabilities of developers. Anshul Bhide, BizOps, India Head at Replit, said in an interview with AIM, “ [We can] make 1x developers into 10x developers with the help of all our compute capabilities, all of our storage capabilities, all of our hosting and deployment capabilities, and all that with obviously Ghostwriter on top of it. That’s what we see.”

According to research conducted by GitHub Copilot, the AI pair programmer makes 88% of programmers feel more productive, with 96% stating that they are faster with repetitive tasks. In fact, almost 3/4ths of developers using the software stated that Copilot helped them stay in the flow, and those using the pair programmer completed tasks 55% more than those who didn’t. 

Even though developers spend time debugging the occasionally buggy code that comes out of the AI tool, it results in a smoother workflow overall. In fact, debugging is probably the biggest disruptor in this workflow. Echoing this sentiment, Xinyun Chen, a Research Scientist at Google Brain and one of the researchers on the paper, stated that improving self-debugging capability is one of the most important things for enhancing LLM coding. 

When combined with other approaches like constitutional AI, Reflexion, AutoGPT, and more, it is possible that by this time next year an average DevOps workflow would look extremely different. In a tweet, Itamar Friedman, the co-founder and CEO of Codium, stated, “Most real-world software products, including “code-gen” tools, will become fully-fledged solutions with stacked modules and AI Agents”. This was said in reference to ‘stacking’ AI models, where AI models invoke other AI models to simulate a sort of emergent intelligence. 

While the argument can be made that this would cause a lot of disruption to developer jobs, it seems that those in the industry don’t have the same thought. Anshul Bhide stated, “There’s going to be some impact on developers. I think the median stack of developers are going to become super charged. I would say overall it’s a productive decade, it’s not going to take away developer jobs.”