Key takeaways
- Criticize AI search engines for prompting LLMs to hallucinate rather than delivering accurate answers.
- Note Bard's recent success over GPT-4, raising expectations for future models like Gemini Ultra.
- Highlight ongoing debates about the appropriateness of using LLMs for search engine functionality.
- Emphasize the importance of real-time information retrieval and proper source attribution for credibility.
- Challenge the fundamental purpose of LLMs, asserting they shouldn't be used merely as search engines.
Google searching the internet with Bard and giving summarised information is sad. Now, there are startups, such as Perplexity, that are selling their bots with this as the primary offering – making it even more depressing.
Recently, Bard, powered by Gemini Pro, overtook OpenAI’s GPT-4 on the Hugging Face Chat Bot Arena Leaderboard. Interestingly, Gemini Pro is only the second-best model by Google in the Gemini series, which increases expectations from the upcoming Gemini Ultra. Adding to this news, a discussion on X questioned if it was right to benchmark LLMs with search engines against raw LLMs.
LLMs for search, really?
Undoubtedly, the problems with LLMs and their hallucinations is something that has been brought up a thousand times earlier, as it hinders the ability of LLM outputs from being taken seriously and can be a major case for misinformation. Connecting them with the web makes sense to retrieve real-time information while also giving proper attribution to the source of information for credibility.
But, this approach also takes away from the main reason that LLMs were built for. “Perplexity is the classic case of misunderstanding what LLMs are made for,” said Subbarao Kambhampati, AI professor at Arizona State University while speaking with AIM. “I am against LLMs as search engines,” he explained that these search engines basically search a couple of links and try to summarise them, but originally they did not start like that.
“AI-based search engines are like prompting LLMs to hallucinate till they give the correct answer. But that depends on whether you know what the correct answer is,” Kambhampati laughed that sometimes even after getting the results, it is important to go back and check whether the answer is correct or not, which negates the whole purpose of the summary in the first place.
Kambhampati makes the case that there is very little information that is left out of models such as GPT-4, barring the recent news of yesterday, he says. “It’s still a useful thing [AI search engines], but it is just a RAG on Google Search and you don’t know where the tokens are coming from,” he added, saying that not every page on Google is prioritised the same way as one would expect.
“Essentially, LLM search is searching by imagination, which is a pretty bad way of searching,” Kambhampati said, while acknowledging that though searching reduces hallucinations in LLMs, relying on the information generated by the LLM connected to search might be risky.
To add another caveat, the NYT and OpenAI lawsuit also highlights the problem with retrieving information from the web as the model makers can be sued for diverting traffic from the original website.
Can’t replace Google Search
The problem that several people point out is that LLM search is basically conversational search with just reference to some blue links. “Conversational search actually relies on results from search engines in the first place. It will augment, but NOT REPLACE, old search engines. RAG is a lot of R and a bit of G,” said Yangqing Jia, founder of Lepton.AI, in a post on X.
However, when it comes to Google, it is arguably going through “the innovator’s dilemma”, which means that alternatives to Google Search are increasing, and though the company is trying to push through, it might be moving away from what it actually offers, which is providing accurate responses to questions with reference.
Dare Obasanjo points out on X that web search is great when you want links to the websites, but LLMs are for answering questions. “LLM search doesn’t replace Google Search, it replaces your need to go to Google for answers,” he added, also adding a joke that if your query could be easily answered by an ad, go to Google, which honestly sounds like a pretty good business idea for Google.
At the end of the day, who knows search better than Google? That explains why it is still running its AI features like an experiment, and not a full fledged search product. It even has the potential to kill startups like Perplexity and alike coming up with LLM search engines with the click of a button – by simply adding an LLM search in its browser, which it did recently on Chrome.
