NVIDIA Releases Garak to Safeguard LLMs

NVIDIA has launched Garak, an open-source vulnerability scanner designed to identify potential weaknesses in LLMs. Dubbed the “nmap for LLMs,” Garak acts as a red-teaming and assessment tool for generative AI systems.

Click here to check out the GitHub repository.

Possibly named after Elim Garak from Star Trek, who is an exiled spy from the Cardassian Union, NVIDIA’s Garak also performs similar tasks like the fictional character. It evaluates LLMs for vulnerabilities such as hallucinations, data leaks, prompt injections, misinformation, toxicity, and jailbreak scenarios.

Garak employs static, dynamic, and adaptive probing techniques to simulate failure modes in AI models and dialogue systems. The tool is free to use, and NVIDIA is actively enhancing its features to support a wider range of applications.

Currently, Garak supports Hugging Face Hub generative models, replicate text models, and OpenAI API chat and continuation models. It also supports anything accessible via the REST API.

“While most of the recent LLMs, especially commercial ones, are aligned to be safer to use, you should bear in mind that any LLM-powered application is prone to a wide range of attacks,” said NVIDIA in the release post.

Last month, NVIDIA launched the Nemotron-4-Mini-Hindi-4B model, a small language model for Hindi, enabling businesses to deploy AI solutions specific to local needs. This model, part of NVIDIA’s NIM microservice, can be deployed on NVIDIA GPU-accelerated systems, optimising performance for various applications.

The chip giant also introduced HOVER (humanoid versatile controller), a 1.5 million parameter neural network designed to coordinate the motors of humanoid robots for locomotion and manipulation.