Guide To Question-Answering System With T5 Transformer

Question Answering is a classical Natural Language Processing. This is a task involving a question being asked to a system from a set of documents or text and should be able to answer that question.

As history has proven, computer science has always helped us in making our lives easier. With the assistance of web search engines, we can get any information at our fingertips. Believe it or not, but the world is trying to make computers act more intelligent. However, one of the key barriers faced by researchers and pioneers is Natural Language Understanding

Question Answering is a classical Natural Language Processing. This is a task involving a question being asked to a system from a set of documents or text and should be able to answer that question. We have seen this application countless times in chatbots on websites. These are so good that many times you cannot tell if it’s a chatbot or a person.  

There are numerous examples of trivia bots that act like quizzing opponents; trivia is a general knowledge question answering the test. Chatbots have to pass the Turing test, which involves a chatbot on one side and a human on the other. The human doesn’t know who is on the other side and to tell if there is a chatbot or a person like them. If the human interpreter fails to do so and there is, in fact, a chatbot at work on the other side, it is a win-win for the chatbot. 


Sign up for your weekly dose of what's up in emerging technology.

Question Answering is very much dependent on a well n good corpus or dataset, you may say. It makes sense to have larger collection sizes which generally lend to better question answering performance until and unless the question domain is orthogonal or quite different from the corpus. Some may think that data redundancy might be a problem because of creating a corpus from the web. Nuggets (small baskets of similar data) of information are likely to be formed in various ways and contexts, greatly benefiting.

Here in this article, we’ll be making a Question-Answering system using T5 Transformer, a state-of-the-art Text to Text transformer developed by Google AI. This transformer has many features and is already trained on the C4 data set (Colossal Clean Common Crawl), around 750 Gigabytes of a text corpus.  You may read about this T5 transformer here in one of my articles.

Download our Mobile App

Let’s dive into the code implementation for making a Q/A system. This will be implemented in a Google Colab notebook whose link will be given below.

Code Implementation of Question Answering with T5 Transformer

Importing Libraries and Dependencies

Make sure the GPU is on in the runtime, that too at the start of the notebook, else it will restart all cells again. If not, then follow this.

Runtime -> Change Runtime -> GPU.

 # check for the GPU provided in the runtime
 # using quiet method for controlling the log
 # for suppressing the colored errors and warning in the terminal
 !pip install --quiet transformers==4.1.1
 # pytorch lightning for smoother model training and data loading
 !pip install --quiet 
 # using HuggingFace tokenizers
 !pip install --quiet tokenizers==0.9.4
 # Google's sentencepiece
 !pip install --quiet sentencepiece==0.1.94
 # mostly pl is used while doing complex model training
 import pytorch_lightning as pl
 # argparse makes it easier to write user friendly command line interfaces
 import argparse
 # package for faster file name matching
 import glob
 # makiing directories for data 
 import os
 # reading json files as the data is present in json files
 import json
 # time module for calculating the model runtime
 import time
 # Allows writing status messages to a file
 import logging
 # generate random float numbers uniformly
 import random
 # regex module for text 
 import re
 # module provides various functions which work on 
 # iterators too produce complex iterators
 from itertools import chain
 from string import punctuation
 # pandas for data manipulation
 import pandas as pd
 # numpy for array operations
 import numpy as np
 # PyTorch
 import torch
 # provides various classes representing file system paths
 # with appropriate semantics
 from pathlib import Path
 from import Dataset, DataLoader
 import pytorch_lightning as pl
 # splitting the data 
 from sklearn.model_selection import train_test_split
 # ANSII color formatting for ouput in terminal
 from termcolor import colored
 # wrapping paragraphs into string
 import textwrap
 # model checkpoints in pretrained model
 from pytorch_lightning.callbacks import ModelCheckpoint
 optimizer - AdamW
 T5 Conditional Generator in which we'll give conditions
 T5 tokenizer because it is fast
 training the model without a learning rate
 from transformers import (
 # Seeds all the processes including numpy torch and other imported modules.
 # check the version provided by Lightning
 import pytorch_lightning as pl
Downloading the Dataset
 # QA dataset from
 # which is in Zip format
 !gdown --id 1mxVUywvKzvA9bvrUc11RYuOTy7MYcXHF
 # Unzipping the folder
 !unzip -q
 # let's have a look at one of the files
 with Path("BioASQ/BioASQ-train-factoid-4b.json").open() as json_file:
   data = json.load(json_file)     
 # Data is a dictionary
 Let’s have a look at how the data is stored and in what format.
 # len of each file
 # We have a list of dictionaries in the "data". We can explore the 0th element
 questions = data['data'][0]['paragraphs']
 # datapoint sample
Function to Create Pandas Dataframes of Questions and Answers

This function will help us read the data from the folder containing multiple JSON files and read them to a dataframe to run manipulation on it and proceed further. 

 def extract_questions_and_answers(factoid_path = Path):
   with as json_file:
     data = json.load(json_file)
     questions = data['data'][0]['paragraphs']
     data_rows = []
     for question in questions:
       context = question['context']
       for question_and_answers in question['qas']:
         question = question_and_answers['question']
         answers = question_and_answers['answers']
         for answer in answers:
           answer_text = answer['text']
           answer_start = answer['answer_start']
           answer_end = answer['answer_start'] + len(answer_text)  #Gets the end index of each answer in the paragraph
                 "question" : question,
                 "context"  : context,
                 "answer_text" : answer_text,
                 "answer_start" : answer_start,
                 "answer_end" : answer_end
     return pd.DataFrame(data_rows)
 factoid_path = Path("BioASQ/BioASQ-train-factoid-4b.json")
 factoid_paths = sorted(list(Path('BioASQ/').glob('BioASQ-train-*')))
 dfs = []
 for factoid_path in factoid_paths:
   df = extract_questions_and_answers(factoid_path)
 df = pd.concat(dfs)
 dfs = []
 # Dropping all the rows with repeated context and questions pairs.
 df = df.drop_duplicates(subset=["context"]).reset_index(drop=True)
 sample_question = df.iloc[243]
 # Using textcolor to visualize the answer within the context
 def color_answer(question):
   answer_start, answer_end = question["answer_start"],question["answer_end"]
   context = question['context']
   return  colored(context[:answer_start], "white") + \
     colored(context[answer_start:answer_end + 1], "green") + \
     colored(context[answer_end+1:], "white")
 print("Answer: ")
 for wrap in textwrap.wrap(color_answer(sample_question), width = 100):

In the following cells, we have instantiated the model and called its tokenizer. T5 tokenizer is pretty fast as compared to other BERT type tokenizers. We will run a sample of this on the text given below and do the decoding.

 # using the base T5 model having 222M params
 MODEL_NAME ='t5-base' 
 tokenizer = T5Tokenizer.from_pretrained(MODEL_NAME)
 sample_encoding = tokenizer('is the glass half empty or half full?', 'It depends on the initial state of the glass. If the glass starts out empty and liquid is added until it is half full, it is half full. If the glass starts out full and liquid is removed until it is half empty, it is half empty.')
 print(len(sample_encoding['input_ids']), len(sample_encoding['attention_mask']))
 # Checking the decoding of the input ids
 preds = [
          tokenizer.decode(input_id, skip_special_tokens=True, clean_up_tokenization_spaces=True)
          for input_id in sample_encoding['input_ids']
 preds= " ".join(preds)
 for wrap in textwrap.wrap(preds, width = 80):
 encoding = tokenizer(
 tokenizer.eos_token, tokenizer.eos_token_id
 # Input id of 1 represents end of sequence token.
 # Text representation pf the input ids
Creating labels for the answers

In the following cells, We have to create necessary labels for the answers. This is required so that we can extract answers accordingly to the questions.

 answer_encoding = tokenizer(
 labels = answer_encoding["input_ids"]
 labels[labels == 0] = -100
Create dataset

In the following cells, we have created the dataset for input in the model. This required setting the lengths, padding and tokenizer. The data has been taken for a BioQA dataset which is specifically for this task. 

 class BioQADataset(Dataset):
   def __init__(
       source_max_token_len: int = 396,
       target_max_token_len: int = 32,
       ): =  data
     self.tokenizer =  tokenizer
     self.source_max_token_len =  source_max_token_len
     self.target_max_token_len =  target_max_token_len
   def __len__(self):
     return len(
   def __getitem__(self, index: int):
     data_row =[index]
     source_encoding = tokenizer(
     target_encoding = tokenizer(
     labels = target_encoding['input_ids']
     labels[labels==0] = -100
     return dict(
 sample_dataset = BioQADataset(df, tokenizer)
 for data in sample_dataset:
   print("Question: ", data['question'])
   print("Answer text: ", data['answer_text'])
   print("Input_ids: ", data['input_ids'][:10])
   print("Labels: ", data['labels'][:10])
Split Dataset into Train and Test

In the following cells, we have split the data into two parts. Test size is small due to the heavy model needed large data for training.

 train_df, val_df = train_test_split(df, test_size=0.05)
 train_df.shape,  val_df.shape 
Create Pytorch Lightning Module

In the following cells, we have created the lightning module. This is made when we have complex models. Hence Pytorch has introduced this to smoothen the process.

 class BioDataModule(pl.LightningDataModule):
   def __init__(
       train_df: pd.DataFrame,
       test_df: pd.DataFrame,
       batch_size: int = 8,
       source_max_token_len: int = 396,
       target_max_token_len: int = 32,
     self.train_df = train_df
     self.test_df = test_df
     self.tokenizer = tokenizer
     self.batch_size = batch_size
     self.source_max_token_len = source_max_token_len
     self.target_max_token_len = target_max_token_len
   def setup(self):
     self.train_dataset = BioQADataset(
     self.test_dataset = BioQADataset(
   def train_dataloader(self):
     return DataLoader(
   def val_dataloader(self):
     return DataLoader(
   def test_dataloader(self):
     return DataLoader(
 data_module = BioDataModule(train_df, val_df, tokenizer, batch_size=BATCH_SIZE)
Loading and Fine Tuning T5

In the following cells, we have installed the model and fine tuned it to our application requirements. 

 model = T5ForConditionalGeneration.from_pretrained(MODEL_NAME, return_dict = True)
 # To check the translation from English to German built-in task 
 input_ids_translated = tokenizer(
     "translate English to German : Oppertunity did not knock until I built a door",
     return_tensors = 'pt'
 generated_ids = model.generate(input_ids = input_ids_translated)
 pred_translated = [
          tokenizer.decode(gen_id, skip_special_tokens=True, clean_up_tokenization_spaces=True)
          for gen_id in generated_ids
 input_ids_summary = tokenizer(
     return_tensors = 'pt'
 generated_ids_summary = model.generate(input_ids = input_ids_summary)
 pred_summary = [
          tokenizer.decode(gen_id, skip_special_tokens=True, clean_up_tokenization_spaces=True)
          for gen_id in generated_ids
 " ".join(pred_summary)
 # Model config
Building the PyTorch lightning module using T5ForConditionalGeneration model

In the following cells, we have leveraged a T5 Conditional Generator which will produce text based on some condition (this is going to be our application task – QA).

 class BioQAModel(pl.LightningModule):
   def __init__(self):
     self.model = T5ForConditionalGeneration.from_pretrained(MODEL_NAME, return_dict=True)
   def forward(self, input_ids, attention_mask, labels=None):
     output = self.model(
     return output.loss, output.logits
   def training_step(self, batch, batch_idx):
     input_ids = batch['input_ids']
     labels = batch['labels']
     loss, outputs = self(input_ids, attention_mask, labels)
     self.log("train_loss", loss, prog_bar=True, logger=True)
     return {"loss": loss, "predictions":outputs, "labels": labels}
   def validation_step(self, batch, batch_idx):
     input_ids = batch['input_ids']
     labels = batch['labels']
     loss, outputs = self(input_ids, attention_mask, labels)
     self.log("val_loss", loss, prog_bar=True, logger=True)
     return loss
   def test_step(self, batch, batch_idx):
     input_ids = batch['input_ids']
     labels = batch['labels']
     loss, outputs = self(input_ids, attention_mask, labels)
     self.log("test_loss", loss, prog_bar=True, logger=True)
     return loss
   def configure_optimizers(self):
     optimizer = AdamW(self.parameters(), lr=0.0001)
     return optimizer
 model = BioQAModel()  
Using Trainer from PyTorch-Lightning to Finetune Model Using our Dataset

We have used a trainer module to test the model and fine-tune it in the following cells. This is helpful when the model is huge, and you want to get a warmup of the model without changing the learning rate. 

 model = BioQAModel()
 # To record the best performing model using checkpoint
 checkpoint_callback = ModelCheckpoint(
 #logger = TensorBoardLogger("training-logs", name="bio-qa")
 #logger = TensorBoardLogger("training-logs", name="bio-qa")
 trainer = pl.Trainer(
     #logger = logger,
     progress_bar_refresh_rate = 30
Loading Tensorboard

In the following cells, we have used a Tensorboard for seeing how the model is progressing against time and epochs. This is very helpful when running huge models. 

 %load_ext tensorboard
 %tensorboard --logdir ./lightning_logs
 #!rm --rf lightning_logs, data_module)
 trainer.test()  # evaluate the model according to the last checkpoint 
 trained_model = BioQAModel.load_from_checkpoint("checkpoints/best-checkpoint.ckpt")
 trained_model.freeze() #  
Generate Answers for the Questions in the Validation Set

In the following cells, we have tried and tested some questions whose answers were saved with us in the valid dataset against the predicted values given by the model. As you’ll notice the T5 model is so powerful that it will predict exact values.

 def generate_answer(question):
       max_length = 396,
   generated_ids = trained_model.model.generate(
       num_beams=1,  # greedy search
   preds = [
           tokenizer.decode(generated_id, skip_special_tokens=True, clean_up_tokenization_spaces=True)
           for generated_id in generated_ids
   return "".join(preds)
 sample_question = val_df.iloc[20]
 sample_question["answer_text"]  # Label Answer
 generate_answer(sample_question)  # Predicted answer
 sample_question = val_df.iloc[66]
 #mkdir zip
 !zip -r / /content
 from google.colab import files"/") 


We have above successfully implemented a Question-Answering system using T5 Transformer and Pytorch Lightning. I highly recommend using quiz data to fine-tune this model to make a trivia bot. Datasets for this specific task are provided here, which have been taken from competitions.


More Great AIM Stories

Mudit Rustagi
Mudit is experienced in machine learning and deep learning. He is an undergraduate in Mechatronics and worked as a team lead (ML team) for several Projects. He has a strong interest in doing SOTA ML projects and writing blogs on data science and machine learning.

AIM Upcoming Events

Regular Passes expire on 3rd Mar

Conference, in-person (Bangalore)
Rising 2023 | Women in Tech Conference
16-17th Mar, 2023

Early Bird Passes expire on 17th Feb

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
27-28th Apr, 2023

Conference, Virtual
Deep Learning DevCon 2023
27 May, 2023

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox

What went wrong with Meta?

Many users have opted out of Facebook and other applications tracking their activities now that they must explicitly ask for permission.