|
Listen to this story
|
From food recipes to medical solutions to even Jesus, there literally exists one or the other versions of ChatGPT. Adding to the band is Microsoft’s BioGPT, a generative pre-trained Transformer language model in a biomedical domain. It can be used for lifescience literature text generation and mining.
The team studied the prompt design and target sequence design when applying BioGPT to downstream tasks and found that target sequences with natural language semantics are better than structured prompts explored in previous works.
The team designed and examined the prompt and the target sequence format while applying pre-trained BioGPT to downstream tasks based on GPT-2 and pre-trained on 15 million PubMed abstracts corpus. It performs better than earlier models on most of the six biomedical NLP tasks it evaluates.
The pre-trained model was used for biomedical NLP tasks such as question answering, document classification, text generation, and end-to-end relation extraction.
BioGPT achieves SOTA on three end-to-end relation extraction tasks and one question-answering task Additionally, it outperforms GPT-2 on the text generation task in terms of lifescience text-generating ability.
It sets a new record with F1 scores of 44.98 percent, 38.42 percent, and 40.76 percent on the BC5CDR, KD-DTI, and DDI end-to-end relation extraction tasks, respectively, and 78.2 percent accuracy on PubMedQA.
On PubMedQA, the larger model ‘BioGPT-Large’ obtains 81.0 percent accuracy. their case study on text generation further highlights the benefit of using BioGPT to create easy definitions for biological terms.
Read the full paper here. Check out the source code here.
Why Does It Matter?
Pre-trained language models have gained significant attention in the biomedical field due to their success in the general natural language domain. Within this domain, there are two main branches of pre-trained language models: BERT (and its variants) and GPT (and its variants). BioBERT and PubMedBERT are examples of the first branch that have received the most attention in the biomedical field. While they have performed well in many discriminative downstream biomedical tasks, their use is limited by their inability to generate text.
Microsoft’s Healthcare Initiatives
In March 2022, Apollo launched ProHealthDeepX, which shows how a patient’s risk factors for heart disease affect their heart through an immersive mixed reality experience using the Microsoft HoloLens 2.
At the leadership summit 2022 in India, Microsoft CEO Satya Nadella spoke about the tech giant partnering with Apollo to develop a cardiac prognosis model that will be trained on data from South Asians.
Microsoft is also working closely with the OpenAI team to employ GPT-3 to facilitate collaboration between employees and clinicians and improve healthcare teams’ efficiency.
