Article complete

Google Introduces Visual Language Maps For Efficient Navigation Using Natural Language

VLMaps merges pretrained visual-language features with spatial mapping of the real world to replicate precision of classic geometric maps.

A crowd gathers in front of a Google booth featuring the company's logo and colorful branding at a tech event.

Key takeaways

  • Google has developed Visual Language Maps, combining visual-language features with spatial mapping for precise navigation.
  • VLMaps can be created using standard exploration techniques from robot video feeds without needing extra labelled data.
  • The system uses a visual-language model to generate pixel-level representations, enhancing the mapping of real-world environments.
  • Landmark names are encoded in an open-vocabulary format and aligned with VLMap pixels through cosine similarity calculations.
  • Large language models like GPT-3 assist in generating executable navigation code from natural language instructions.

Takeaways drafted with AI from the article text.

Standard exploration techniques can be used to create VLMaps from robot video feeds without any additional labeled data, and these maps facilitate natural language indexing.

This is accomplished by utilising a visual-language model to generate rich pixel-level representations from the RGB-D video feed of the robot, and then projecting these representations onto the 3D surface of the environment. This surface is captured through depth data used for reconstruction using visual odometry.

Ultimately, the scene's top-down representation is created by saving the visual-language characteristics of every image pixel at the corresponding location of the grid map pixel.

How VLMaps navigates & processes natural language 

Firstly, the text encoders within the Visual Language Model are used to encode the names of landmarks in an open-vocabulary format, such as "chair", "green plant", and "table". Next, these landmark names are aligned with the pixels in the VLMap by calculating the cosine similarity between their embeddings. By performing an argmax operation on the similarity score, the mask for each type of landmark is obtained.

Additionally, large language models are used to create navigation policies in the form of code that can be executed. To achieve this, GPT-3 is utilised. It is provided with a few examples as prompts. GPT-3 parses the language instructions and produces a string of executable code, which includes expressions of functions or logical structures and API calls with parameterisation (e.g., robot.move_to(target_name) or robot.turn(degrees)).

Using VLMaps between multiple robots

VLMaps facilitate natural language-based, long-horizon spatial goal navigation by using open-vocabulary landmark indexing. 

Sharing a VLMap between multiple robots allows for the generation of obstacle maps specific to different embodiments in real-time, which improves navigation efficiency. For instance, while a ground robot must avoid obstacles, a drone can ignore them. 

The team conducted experiments to demonstrate how a single VLMap representation in each scene can adjust to different embodiments by creating custom obstacle maps and enhancing navigation efficiency.

Published

Spotted an error? Tell our editors. Read our editorial standards and corrections.

About the author

Contributor

All stories
Belamy · Daily

The day’s AI news, before your first meeting.