Article complete

NVIDIA’s NVILA Beats GPT-4o Mini and Llama 3.2, Redefines Open Standards in Vision AI

NVIDIA is at it again, this time with a segment-leading vision model that reads images, and videos as input

A speaker in a black jacket gestures with glowing green hands, against a backdrop of server racks and digital patterns.

Key takeaways

  • NVIDIA's NVILA model outperforms competitors like GPT-4o Mini and Llama 3.2 in vision benchmarks.
  • Achieves 4.5x lower training costs and 3.4x reduced fine-tuning memory compared to LLaVa OneVision.
  • Implements 'scale then compress' technique for balancing accuracy and efficiency in visual language models.
  • NVIDIA plans to release NVILA code and models for reproducibility on Hugging Face soon.
  • Training a 7B VLM requires 400 GPU days and over 64GB of GPU memory.

Takeaways drafted with AI from the article text.

Recently, NVIDIA released a new family of open visual language models called NVILA, which focuses on optimised accuracy and efficiency. The model is said to reduce training costs by 4.5 times and fine-tuning memory by 3.4 times. Moreover, it also reduced latency for pre-filling and decoding by nearly 2 times. The numbers are all in comparison with the LLaVa OneVision model, which is another large vision model coupled with the Qwen 2 language model. 

From the benchmark results, NVILA is better than GPT 4o Mini in a video benchmark and offers competitive performance compared with GPT 4o, Sonnet 3.5, and Gemini 1.5 Pro. That said, it was observed that NVILA offered better performance than most open models. It also scored a marginal victory over Llama's 3.2 model.

However, it seems that they haven’t released the model on Hugging Face yet. "We will soon make our code and models available to facilitate reproducibility," said NVIDIA.

NVIDIA said that training a visual language model (VLM) is expensive, and it takes around 400 GPU days to train a 7B VLM. Moreover, they also stated that fine-tuning a VLM is ‘memory intensive’, and a 7B VLM consumes over 64GB of GPU memory. 

Therefore, NVIDIA is using a technique called ‘scale then compress’, an approach to balancing accuracy and efficiency in VLMs. Instead of reducing the size of photos and videos, NVILA uses high-resolution images and multiple frames from a video to ensure that no details are lost. 

Then, the model reduces the size of the inputs by squeezing visual information into fewer tokens, grouping the pixels together, and retaining important information.

“For example, doubling the resolution will double the number of visual tokens, which will increase both training and inference costs by more than 2×, as self-attention scales quadratically with the number of tokens. We can then cut this cost down by compressing spatial/temporal tokens,” said the authors in the paper detailing the model.

NVIDIA also outlined a few demos from the model, and it was able to provide information from multiple queries based on an image, and a video. The outputs were also compared to the VILA 1.5 model, also previously released by NVIDIA. 

NVIDIA also detailed using other techniques, like Dynamic-S2 for scaling, DeltaLoss-based dataset pruning, quantisation using FP8 precision, and more. The paper can be read on Arxiv for a breakdown of how these techniques aid the model. All of these techniques were applied to an 8B parameter model. 

Published

Spotted an error? Tell our editors. Read our editorial standards and corrections.

More in this story

AI (Artificial Intelligence) · 5 stories
  1. You are here: NVIDIA’s NVILA Beats GPT-4o Mini and Llama 3.2, Redefines Open Standards in Vision AI
  2. Investor With No Coding Experience Builds ‘$1.7Mn SaaS’ Using Replit 
  3. Perplexity Launches Sonar for Pro Users; Performance on Par with GPT-4o, Claude 3.5 Sonnet
  4. Google Cloud, Liberty Global Enter a 5-Year AI Partnership
  5. Only 35% Indian Enterprises are Prepared for AI Cyberattacks: Kyndryl Study
About the author

Journalist

All stories
Belamy · Daily

The day’s AI news, before your first meeting.