Tech Behind Facebook’s BlenderBot 2.0, A Chatbot With Long Term Memory

During a dialogue, the model combines an information retrieval component with a text generator.

BlenderBot 2 is a first of its kind open-source chatbot with long term memory– from the house of Facebook’s research platform ParlAI. The latest developments point to Facebook’s bid to make the AI more empathetic, knowledgeable and capable.

Language models like OpenAI’s GPT 3, and BlenderBot 1.0 are quite articulate and generate human-like text. BlenderBot was infamous for having the memory of a goldfish and its tendency to feign knowledge. Also the long-term memory of such systems is static– ie limited to what the models have been taught. 

Credits: Facebook AI Blog Post

AIM Daily XO

Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

All about BlenderBot 2.0

The BlenderBot 2.0 can build long term memory for continuous access. Moreover, it can do so while simultaneously searching for information on the internet and holding conversations on nearly any topic. So, if a user talked about Tom Brady weeks ago, BlenderBot 2.0 will likely bring up the NFL in future conversations. 

In conversational experiments, BlenderBot 2.0 generated contextual internet searches via Bing to respond with information and hold longer and factually correct conversations. The chatbot simultaneously stored conversational information in its LTM to leverage that knowledge in more extended ongoing discussions. To ensure that no data is passed between users, information is stored differently for different people. 

Download our Mobile App

“While existing systems can ask and answer basic questions about things like food, movies, or bands, they typically struggle with more complex or freeform conversations, like, for example, discussing Tom Brady’s career in detail”, Facebook research scientist Jason Weston and research engineer Kurt Shuster wrote.“But technology-based on BlenderBot 2.0 could one day become a useful part of everyday life by being able to have multisession conversations on any topic that can last days, weeks, or even months, and by adding to what it knows and can talk about as the conversation evolves”.

Click here to watch the demonstration video. 

The new system outperformed BlenderBot 1.0 in picking up conversation sessions with a 17 percent improvement in engagingness score, and 55 percent improvement in use of previous conversation sessions. The rate of hallucination has also dropped from 9 percent to 3 percent. ParlAI claimed BlenderBot 2.0 outperformed the conversational abilities of the existing systems. With its commonsense reasoning capacities, BlenderBot 2.0 can reduce its hallucination and limit its confusion between subtle concepts. 

Tech behind

BlenderBot 2.0 uses an AI model based on Retrieval Augmented Generation, an approach that enables it to generate responses and incorporate knowledge beyond the conversation. 

During conversation, the model seeks relevant information both in its long-term memory and from the internet. The model uses an enhanced traditional encoder-decoder architecture coupled with an additional neural network module to generate relevant search queries. BlenderBot 2.0 then prepends the resulting knowledge to the conversational history encoded using the Fusion-in-Decoder method. The chatbot generates a response based on the encoded knowledge

Credits: Facebook AI Blog Post

ParlAI researchers used a crowdsourcing platform to put together two datasets for training the neural network: Wizard of the Internet contains human conversations augmented with new internet searches. This guides BlenderBot 2.0 on generating search engine queries and creating responses based on it; Multisession has a long-context chat with humans referencing information from conversion history. It directs the bot on which new knowledge to store in long term memory.

Safety features 

Language models run the risk of picking bias from the dataset it’s trained on. Facebook has introduced two prompts to mitigate this issue- baked-in safety and robustness to difficulty. The methods work by training data from the new human-and-model-in-the-loop framework in a two-stage system to ‘bank in’ safety to the generative model itself. 

The research showed a 90 percent reduction in offensive responses and a 74.50 percent increase in safe responses.
BlenderBot 2.0 can continue a conversation as long as information about it is available on the web- it is limited only by what a search engine can provide. In an interview, Jason Weston said that while BlenderBot 2.0 may be able to use information about other languages if provided on Bing, the chatbot is, for now, only focused on English based results.

Sign up for The Deep Learning Podcast

by Vijayalakshmi Anandan

The Deep Learning Curve is a technology-based podcast hosted by Vijayalakshmi Anandan - Video Presenter and Podcaster at Analytics India Magazine. This podcast is the narrator's journey of curiosity and discovery in the world of technology.

Avi Gopani
Avi Gopani is a technology journalist that seeks to analyse industry trends and developments from an interdisciplinary perspective at Analytics India Magazine. Her articles chronicle cultural, political and social stories that are curated with a focus on the evolving technologies of artificial intelligence and data analytics.

Our Upcoming Events

24th Mar, 2023 | Webinar
Women-in-Tech: Are you ready for the Techade

27-28th Apr, 2023 I Bangalore
Data Engineering Summit (DES) 2023

23 Jun, 2023 | Bangalore
MachineCon India 2023 [AI100 Awards]

21 Jul, 2023 | New York
MachineCon USA 2023 [AI100 Awards]

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox