Traditional search engines have existed for over two decades now. Google, which is synonymous with search, launched its search engine in 1998 and over the years, has become the most-used search engine worldwide.

Nothing stirred for years, and search engines continued to work in the same fashion – indexing and extracting on a keyword hit. But, things are about to change. Microsoft is set to change the rules of the game with the integration of ChatGPT with its search engine Bing. 

Although Bing won’t be the first search engine to integrate a large language AI model since You.com, Neeva AI and Perplexity AI have already done it successfully so far, a bigger player initiative would mark the evolution of search engines. 

Traditional search engines have also been using AI for a few years now. Google Search has already been leveraging the power of AI since 2015 when it introduced the RankBrain algorithm. Over the years, other algorithms such as Neural matching, BERT and MUM, among others, have been incorporated into Google Search.

However, recent approaches have been very different. Large Language Models (LLMs) are rapidly developing and mastering various applications. Now, the capabilities of LLMs are being used to define the next generation of search engines. 

Rise of reasoning engines

Traditional search engines do not address information needs directly, and instead provide references. Therefore, by leveraging the capabilities of LLMs like GPT-3, startups are building search engines which provide pointed answers to your search query, along with the indexed web pages. 

Privacy-focused search engine You.com recently launched youChat, a ChatGPT-like chatbot that provides answers upfront along with citations. You.com has added the bot to its search engine as a feature.

Similarly, Neeva AI, a paid search engine, goes a step further. Once a user types his query, the AI-generated response is the first thing you see.

(Source: Twitter)

“Neeva AI is leading a new frontier of AI search that leverages cutting-edge LLMs and an independent search stack to create a unique and transformative search experience,” the company said.

The extractive nature of traditional search engines and the abstractive nature of the generative approach found in LLMs that underpin ChatGPT have to come to terms with each other to provide a better experience for the user. 

“This will result in a new generation of the search engine – the ‘reasoning engine’ – that can provide a human-like conversational response backed and reasoned by source through citations,” Raj Neervannan, co-founder and CTO, AlphaSense, told AIM. 

Neervannan explained how AlphaSense has been leveraging the latest AI technologies to provide search results in a more natural, native way that is seamless to the user. “We use language models to teach semantics to our search engines and have been combining it with the ability to provide precise and stateful search results that tie the results back to the source at the snippet level in any document or source. For our users conducting market research, this technology helps instantly identify trends and summarise long reports, as well as more easily gather, digest, and analyse large amounts of information across multiple languages,” he added.

What happens to traditional search engines?

We believe traditional search engines like Google Search and Bing will continue to exist and evolve and leverage the capabilities of LLMs. Microsoft is already planning to integrate ChatGPT with Bing. Google, on the other hand, published a research paper titled ‘Rethinking Search: Making Domain Experts out of Dilettantes’  in this regard in 2021. Researchers at Google proposed a radical redesign that throws out the ranking approach and replaces it with a single large AI language model—a future version of BERT or GPT-3. 

This approach not only changes how search engines work but also how we interact with them. “Traditional search engines will have to learn to rank better with a stateful engine in a native way, and generate a response that summarises the search results. This is a big jump from the current models that just index and extract on a keyword hit,” Neervannan concluded.