Did ChatGPT Just Kill Stack Overflow?

Stack Overflow, the Q&A platform for developers has always been the go to place for all programmers. It is a massive community for discussion and collaboration. In December last year, the forum decided to ban posting information generated by ChatGPT, citing high inaccuracy in the answers that the bot provides and how that can be “substantially harmful to the sites and the users looking for correct answers”.

However, this ban on ChatGPT did not bode that well for Stack Overflow. According to reports from SimilarWeb, after the release of ChatGPT in November, there was a 12% decrease in the number of website visits, from 279 million to 247 million in December. After just a little rise in January to 249 million, the website visits dropped even lower in February, to 239 million. 

The Stack Overflow team had told AIM that the drop in the number of visits in December was only because of the holiday season and nothing else. When it comes to the drop in February, the team told AIM that it is because there are less number of days in the month, and the number actually increased if we count daily users. Even if that holds true, the report from SimilarWeb does bring up the question -Whether people are moving to ChatGPT over Stack Overflow or not? And Why might that be the case?

ChatGPT is Too Convenient

Developers have been copying code from Stack Overflow all these years. ChatGPT now makes them believe that it’s original, even if it’s not. It is too convenient for developers to post their questions on ChatGPT and get immediate responses instead of going to Stack Overflow and explaining every single step of the problem. 

The interesting relationship between ChatGPT and Stack Overflow is that the LLM-based chatbot is trained on the data available on Stack Overflow. This is confirmed by the fact that the GPT-3 paper mentions that it is trained on multiple datasets, including Common Crawl, which essentially means everything on the internet. Unless OpenAI didn’t actively take steps to remove Stack Overflow from the training data, there is no reason to believe that it is not included. 

This makes ChatGPT a very convenient tool for developers. Instead of browsing through Stack Overflow for hours to find the perfect answer matching their query, ChatGPT can give out the perfect solution since it is based on the data from Stack Overflow anyway.

However, ChatGPT’s system is not fool-proof and is filled with wrong, useless, and thousands of answers, much like Stack Overflow. So, if ChatGPT is trained on all of Stack Overflow’s data—and not only the most upvoted answers—it is bound to give out false answers as well. Such has been the case several times, eventually leading users to head back to the internet. It is definitely not wise to trust the chatbot’s answers too much.

But OpenAI took another big step recently to overcome the issue of unreliability. The launch of the Code interpreter plugin on ChatGPT has made it possible for developers to verify their own code on the chatbot. Currently, this plugin is only available on ChatGPT Plus, which runs on GPT-4. 

Apart from ChatGPT, Bing Chat also enables developers to access the internet. This allows them to move beyond the 2021 dataset cut-off of ChatGPT, and incorporate code with up-to-date information. 

Stack Overflow’s Overflowing Community

Stack Overflow is the community of developers and that makes the challenge difficult for ChatGPT. Though a lot of people have stopped visiting Stack Overflow completely since the launch of ChatGPT, the ability to discuss answers is still something that ChatGPT, or any other chatbot, can never fulfil. 

According to the stats from SimilarWeb, though there has been a dip in the traffic of Stack Overflow, it is not that monumental. Developers believe that the community and, more importantly, human interaction is important for coding as well. Solutions provided by expert developers on the forum seem to be more reliable than ChatGPT’s extremely overconfident writing skills.

Conversely, some developers cite the community as “toxic”. Though it is not right to paint the entire community in the same colour, it is believed by a lot of users that developers go on Stack Overflow to berate others instead of it merely functioning like a Q&A site. 

At present, it is highly unlikely that Stack Overflow would embrace AI tools on its platform. Stack Overflow’s response to banning ChatGPT answers is the same as Getty Images banning AI art on its platform while Shutterstock, its rival, had decided to embrace AI art on its stock image marketplace by compensating the original artists. 

Likewise, there might be a way forward for Stack Overflow as well to deal and adapt to this changing landscape if it wants to. 

For now, developers have decided to take the approach of Docs > GPT > Google Search > Stack Overflow.