Article complete

NVIDIA Introduces ChatQA, GPT-4 Level Conversational QA Models

ChatQA-70B stands out by surpassing GPT-4 in average scores on 10 conversational QA datasets (54.14 vs. 53.90).

NVIDIA Introduces ChatQA, GPT-4 Level Conversational QA Models

Key takeaways

  • NVIDIA's ChatQA-70B model surpasses GPT-4 in conversational QA accuracy on multiple datasets.
  • The model achieves high performance without using synthetic data from ChatGPT.
  • ChatQA utilizes a two-stage instruction tuning method to enhance zero-shot QA results.
  • NVIDIA fine-tunes a single-turn query retriever, achieving competitive results with reduced costs.
  • Incorporating unanswerable samples significantly boosts ChatQA's performance in challenging scenarios.

Takeaways drafted with AI from the article text.

NVIDIA's ChatQA introduces a range of models, ranging from 7B to 70B in size. Extensive evaluations across 10 conversational QA datasets reveal that the top-performing ChatQA-70B model not only outperforms GPT-3.5-turbo but also performs at par with GPT-4. What's notable is that these achievements are attained without relying on any synthetic data from ChatGPT models.

The team behind ChatQA proposes a two-stage instruction tuning method, significantly enhancing zero-shot conversational QA results from large language models (LLMs). To address retrieval in conversational QA, a dense retriever is fine-tuned on a multi-turn QA dataset, delivering comparable results to state-of-the-art query rewriting models with reduced deployment costs.

NVIDIA showcases the effectiveness of fine-tuning a single-turn query retriever using their curated conversational QA data. This approach performs comparably to the state-of-the-art LLM-based query rewriting model, all achieved without the need for extra computational time and potential API costs associated with rewriting.

NVIDIA's ChatQA has demonstrated a remarkable advancement in handling scenarios where answers are elusive. The incorporation of a small number of "unanswerable" samples has proven to significantly enhance the model's capabilities. Through an evaluation of unanswerable cases, it is evident that the leading model, ChatQA-70B, exhibits only a slight performance gap when compared to the formidable GPT-4. 

For more details, refer to the paper: ChatQA Paper.

NVIDIA is not alone. Several foundational models have achieved the capabilities of GPT-4. Google is likely to launch Gemini Ultra at any moment. Meanwhile, Mistral CEO Arthur Mensch announced on French national radio that the company will unveil an open-source GPT-4-level model in 2024. 

Published

Spotted an error? Tell our editors. Read our editorial standards and corrections.

About the author

Tech Journalist

Siddharth is a media graduate who loves to explore tech through journalism and putting forward ideas worth pondering about in the era of artificial intelligence.

All stories
Belamy · Daily

The day’s AI news, before your first meeting.