Google Turns the Mouse Pointer Into an AI Agent With Gemini

Google is rethinking one of the oldest elements in computing, the mouse pointer, by turning it into an AI-powered interface that can understand what users are pointing at and what they want to do with it.

In a blog post published on May 12, Google outlined a new concept for an AI-enabled pointer, powered by Google’s Gemini models.

It lets users interact with content using short voice commands and gestures instead of detailed prompts. In one example, a user points at a building in an image and says, “Show me directions.” 

“The mouse pointer has been a constant companion on computer screens,” the company said. “Despite how technologies have changed, the pointer has barely evolved in more than half a century.”

According to Google, the system can identify text, images, objects, locations and other on-screen elements in real time. The company said this could allow users to summarise PDFs, convert tables into charts, edit images, or turn handwritten notes into actionable lists using simple pointer-based interactions.

We’re reimagining a 50-year-old interface - the mouse pointer - with AI. 🖱️

These experimental demos show how people can intuitively direct Gemini on their screens using motion, speech, and natural shorthand to get things done 🧵 pic.twitter.com/p6fhgNcopz

— Google DeepMind (@GoogleDeepMind) May 12, 2026

Google said current AI tools often interrupt workflows because users must manually copy content into chat windows. The company said its goal is to build AI that works directly inside the apps and interfaces people already use.

“We want the opposite: intuitive AI that meets users across all the tools they use, without interrupting their flow,” the researchers wrote.

Google is already bringing parts of the system into products such as Google Chrome and Googlebook.

 Starting May 12, users can use Gemini in Chrome to ask questions about selected parts of a webpage instead of writing full prompts.

Google is already putting this vision into hardware as well.

The company has also introduced Googlebook, a new category of laptops built around its Gemini AI models, as it pushes to redesign personal computing around AI-powered interactions.

A central feature of the new laptops is Magic Pointer, an AI cursor developed with the Google DeepMind team. Google said the feature allows Gemini to understand what users are pointing at on screen and offer contextual actions.

“Just wiggle your cursor and watch it come alive with Gemini, offering quick, contextual suggestions every time you point at something on your screen,” the company said.

Google said users could point at a date in an email to create a meeting or select two images, such as a living room photo and a couch, to visualise them together.