Guide To SciBERT: A Pre-trained BERT-Based Language Model For Scientific Text

 SciBERT is a pre-trained BERT-based language model for performing scientific tasks in the field of Natural Language Processing (NLP). It was introduced by Iz Beltagy, Kyle Lo and Arman Cohan - researchers at the Allen Institute for Artificial Intelligence (AllenAI) in September 2019 (research paper).

Since the architecture of SciBERT is based on the BERT (Bidirectional Encoder Representations from Transformers) model, go through the BERT research paper if you are unaware of the state-of-the-art base model.

Overview of SciBERT 

Training deep neural networks for NLP tasks requires a huge amount of labelled data. Though it is reasonable to gather such a large amount of data in several general domains, the data acquisition task seems difficult when it comes to scientific domains due to the expertise required for annotating scientific data. SciBERT, a model trained on a large corpus of scientific data, leverages unsupervised pre-training and significantly improves the performance of BERT model in scientific NLP tasks. 

Background of SciBERT

The base model BERT is trained on two tasks:

• Predict randomly masked tokens
• Predict whether two sentences follow each other

SciBERT follows the same model architecture as BERT; the only difference is - it is trained on scientific data instead.

Vocabulary used by SciBERT

The original WordPiece vocabulary used by the BERT model is termed as BASEVOCAB while dealing with SciBERT. Another WordPiece vocabulary used by SCiBERT is called SCIVOCAB which is constructed on scientific corpus using SentencePiece library.

Core NLP tasks SciBERT can accomplish

• Named Entity Recognition (NER): 
• PICO Extraction
• Text Classification (CLS)
• Relation Classification (REL)
• Dependency Parsing (DEP)

Variants of SciBERT

There are four versions of the SciBERT model based on:

(i) cased or uncased

(ii) BASEVOCAB or SCIVOCAB

The two models using BASEVOCAB are fine tuned from the corresponding BERT-base models. The other two models which use SCIVOCAB are trained from scratch.

Practical implementation

Here’s a demonstration of NCBI disease corpus task - a Named Entity Recognition (NER) task in the biomedical field. The data used is a part of a collection of 793 PubMed abstracts having annotated disease entities. Every token entity has a ‘B-’ (Beginning) tag indicating if the token is at the start of the entity or an ‘I-’ (Inside) tag indicating that the token is inside the annotation while the ‘O’ tag suggests that the token is not a named entity.

The code has been implemented in Google colab with Python version 3.7.10. Step-wise explanation of the code is as follows:

1. Get the NCBI data from The AllenAI’ s GitHub repository

 %%bash
 DATADIR="NCBI_disease" #Name given to the directory
 if test ! -d "$DATADIR";then
     echo "Creating $DATADIR dir" #print statement
     mkdir "$DATADIR" #Create a directory if it doesn’t exist
     cd "$DATADIR" #change the current working directory
     
     #download development set data
     wget https://raw.githubusercontent.com/allenai
     /scibert/master/data/ner/NCBI-disease/dev.txt
     
     #download training data
     wget https://raw.githubusercontent.com/allenai
     /scibert/master/data/ner/NCBI-disease/test.txt
     
     #download testset data
     wget https://raw.githubusercontent.com/allenai
     /scibert/master/data/ner/NCBI-disease/train.txt
 fi #end of ‘if’ condition 

2) Clone the GitHub repository of bert-sklearn, a scikit-learn wrapper for fine-tuning the BERT model

!git clone -b master https://github.com/charles9n/bert-sklearn

Change the directory and install bert-sklearn

!cd bert-sklearn; pip install .

3) Import required libraries

 import os
 import math
 import random
 import csv
 import sys
 import numpy as np
 import pandas as pd
 from sklearn import metrics
 from sklearn.metrics import f1_score, precision_score, recall_score
 from sklearn.metrics import classification_report
 import statistics as stats
 from bert_sklearn import BertTokenClassifier 

4) Define a function to read tsv file (‘tsv’ stands for ‘tab-separated values’)

 def read_tsv(fname, quotechar=None):
 #open the utf-8 encoded file in read mode
     with open(fname, "r", encoding='utf-8') as f:
         return list(csv.reader(f, delimiter="\t", quotechar=quotechar))   

csv.reader() function generates a reader object to extract data from a csv file. It takes each line of the file and makes a list of all the columns. 

You then need to choose just the column from which you want the variable data.

5) Define a function to flatten the array of tokens

    def flatten(l):
       return [item for sublist in l for item in sublist] 

6) Define a function to read the data file in CoNLL-2003 shared task format.

  def read_CoNLL2003(fname, index=3):
     # Read the file
     lines =  open(fname).read().strip()   
     # Find sentence-like boundaries by splitting on seeing 2 newline   
     characters.
     lines = lines.split("\n\n")  
     # Split on new lines
     lines = [l.split("\n") for l in lines]
     # Get tokens
     tokens = [[l.split()[0] for l in line] for line in lines]
     # Get output labels or tags
     labels = [[l.split()[index] for l in ln] for ln in lines]
     #Convert the data comprising tokens and corresponding labels to a  
     dataframe
     dt = {'tokens': tokens, 'labels': labels}
     df=pd.DataFrame(data=dt)
     return df  

7) Define a function to read train, dev and test set data 

 DATADIR = "NCBI_disease/"
 def get_data(trainfile=DATADIR + "train.txt",
              devfile=DATADIR + "dev.txt",
              testfile=DATADIR + "test.txt"):
     #Read the train,dev and test files provided as parameters 
     train = read_CoNLL2003_format(trainfile, index=3)    
     dev = read_CoNLL2003_format(devfile, index=3)
     #Combine the train and dev set
     train = pd.concat([train, dev])
     print("Train and dev data: %d sentences, %d tokens"%
     (len(train),len(flatten(train.tokens))))
     test = read_CoNLL2003_format(testfile, index=3)
     print("Test data: %d sentences, %d tokens"%
     (len(test),len(flatten(test.tokens))))
     return train, test 

8) Perform train-test split

 #Store the train and test set data returned by the get_data() function defined in step (7)
  train, test = get_data()
 #Separate out features (tokens) and labels of training and test set
 X_train, y_train = train.tokens, train.labels
 X_test, y_test = test.tokens, test.labels
 print(len(train)) #Print the number of instances in training set
 labels = np.unique(flatten(y_train)) #get unique labels
 labels = list(label_list) #form a list of labels
 print("\nNER tags:",labels) #print the list containing unique labels 

Output:

 Train and dev data: 6347 sentences, 159670 tokens
 Test data: 940 sentences, 24497 tokens
 6347
 NER tags: ['B-Disease', 'I-Disease', 'O'] 

9) See the initial records of training data

   train.head()

10) Initialize the SciBERT model

Out of the four versions of SciBERT, here we are using BASEVOCAB  CASED version.

 %%time  #to record execution time
 model = BertTokenClassifier
 (bert_model='scibert-basevocab-cased',
                             max_seq_length=178, 
                             epochs=3,
       #gradient accumulation
                             gradient_accumulation_steps=4,
                             learning_rate=3e-5,
                             train_batch_size=16,#batch size for training
                             eval_batch_size=16, #batch size for evaluation
                             validation_fraction=0., 
                     #ignore the tokens with label ‘O’                      
                             ignore_label=['O'])  

The ‘max_seq_length’represents the length of a token sequence that the model can handle. BERT’s limit is 512 tokens but here we explicitly limit it to 178 (176 tokens + 2 for [CLS] and [SEP] delimiters used by BERT model)

 #Print the model’s configuration
 print(model) 

Sample output:

11) Fit the fine-tuned BERT model on training data

    model.fit(X_train, y_train)

Sample output:

12) Make predictions on test data

y_preds = model.predict(X_test)

13)Print classification report on model’s performance

print(classification_report(flatten(y_test), flatten(y_preds)))

Sample output:

Note: The outputs may vary a bit for each execution of the code and also depending on the execution environment you choose.

• Code source
• Google colab notebook of the above implementation can be found here.

References

Refer to the following sources to have in-depth understanding of the SciBERT model:

• SciBERT research paper
• BERT research paper 
• bert-sklearn GitHub repository
• SciBERT GitHub repository