Article complete

NVIDIA Releases Garak to Safeguard LLMs

It evaluates LLMs for vulnerabilities such as hallucinations, data leaks, prompt injections, misinformation, toxicity, and jailbreak scenarios.

A humanoid alien character with distinct facial features sits in a futuristic setting, looking directly at the viewer.

Key takeaways

  • NVIDIA launches Garak, an open-source scanner for assessing LLM vulnerabilities.
  • Garak evaluates LLMs for issues like hallucinations, data leaks, and prompt injections.
  • Utilize Garak's static and dynamic probing techniques to simulate AI model failure modes.
  • Garak supports various generative models, including those from Hugging Face and OpenAI.
  • NVIDIA emphasizes that all LLM applications remain susceptible to diverse security threats.

Takeaways drafted with AI from the article text.

Possibly named after Elim Garak from Star Trek, who is an exiled spy from the Cardassian Union, NVIDIA’s Garak also performs similar tasks like the fictional character. It evaluates LLMs for vulnerabilities such as hallucinations, data leaks, prompt injections, misinformation, toxicity, and jailbreak scenarios.

Garak employs static, dynamic, and adaptive probing techniques to simulate failure modes in AI models and dialogue systems. The tool is free to use, and NVIDIA is actively enhancing its features to support a wider range of applications.

Currently, Garak supports Hugging Face Hub generative models, replicate text models, and OpenAI API chat and continuation models. It also supports anything accessible via the REST API.

“While most of the recent LLMs, especially commercial ones, are aligned to be safer to use, you should bear in mind that any LLM-powered application is prone to a wide range of attacks,” said NVIDIA in the release post.

Last month, NVIDIA launched the Nemotron-4-Mini-Hindi-4B model, a small language model for Hindi, enabling businesses to deploy AI solutions specific to local needs. This model, part of NVIDIA’s NIM microservice, can be deployed on NVIDIA GPU-accelerated systems, optimising performance for various applications.

The chip giant also introduced HOVER (humanoid versatile controller), a 1.5 million parameter neural network designed to coordinate the motors of humanoid robots for locomotion and manipulation.

Published

Spotted an error? Tell our editors. Read our editorial standards and corrections.

About the author

Journalist

Mohit writes about AI in simple, explainable, and often funny words. He's especially passionate about chatting with those building AI for Bharat, with the occasional detour into AGI.

All stories
Belamy · Daily

The day’s AI news, before your first meeting.