Meet the Researcher Building Real Robots for Real Household Chores

Without any disagreement, natural language is the most popular segment in AI today. Every month, there is a breakthrough that everyone goes gaga about. If you wonder what is the robotics equivalent of all of this, "there are not many answers", believes Lerrel Pinto, assistant professor of computer science at NYU Courant, working in robotics and machine learning. 

"The progress in robotics is definitely a lot slower. It's one of the harder and more impactful problems for several reasons. That's why we work on robotics," he shared in an interview with AIM. 

In September, Pinto made it to MIT Technology Review's 2023 Innovators Under 35 list. As per Pieter Abbeel, the director of the robot learning lab at the University of California, Pinto's current research will be looked back upon as having laid many of the early building blocks of the future of robot learning. And there's good enough reason to believe so. 

According to the 31-year-old researcher, "Eventually, we'll have robots in our homes doing things that we as humans don't want to do." Following this frame of mind, a month ago, alongside his team, Pinto introduced Dobb-e, an open-source, general framework for robots to learn household manipulation. 

"When you look at most research in robotics, they're all done in these very constrained lab environments," Pinto rightly pointed out. He added that every robotics lab has a white table top and a neat background behind it. You ask the robot just to pick up one object and place it somewhere. You'll find that the most robotic research papers follow this.

The roboticist ain't wrong. If you look at the big tech research labs for robotics, they look exactly how Pinto described it. Earlier this year, Google launched PaLM-E, which can integrate vision and language for robotic control. Here's a picture of the research lab from the company's demo video -- just like Pinto described it. 

"From my viewpoint, you will never solve hard problems until you actually try to solve them. If you're always working in a lab environment, you're never going to make progress on the actual hard problems," Pinto believes. His team did the Dobb-e project "hopefully trying to convince the research community to stop working on toy problems and focus on real problems inside people's homes," he said. 

Generative AI meets Robotics

Pinto dreams of seeing robots in our environment where they are very close to humans, running chores, doing the dishes and laundry or rearranging objects on a table. 

"Now, in the context of these types of problems, generative AI is most useful when you have data," he said. "If I show the robot some data on how to fold a t-shirt and give it a new t-shirt to fold, that is a generative process. It's a process of generating robotic behaviour. Tools in generative AI, like diffusion models and transformers, have been directly used on these problems, and they work well. Our lab has worked on some transformer models, and this field is moving fast," Pinto described.  

But the pain point of generative AI is everyone is trying to do something with it. Pinto explained that there's a bidirectional effect because certain things get popular, and people start doing more topical things. Many people doing niche research start focusing on popular topics, which is a choice people have to make, but there's a lot of exciting work in the community. 

He further highlighted Google DeepMind's work at the intersection of AI and sciences. "These types of problems where one can make the system more efficient with AI and impact people's lives," he noted. 

Choose Data Wisely

The problem of bias in data has been extensively covered since the GPTs gained popularity. With robotics, you have a fascinating problem where most of the data is generated by roboticists because there is no internet of robot data, the researcher explained. "Most of the robot data is created by researchers like me at Google, Berkeley, and other places to show the robot how to do things. There you have a problem because the choice of what types of tasks you show the robots have a huge impact on where these robots will work," Pinto mentioned. 

For example, Pinto said if he showed the robot how to open a door, where the doors are only in affluent houses. Now, if he takes this robot and goes to maybe a less affluent household, the way it looks will be different. The handles will look different, and the robot won't know how to operate in these houses. 

This is the case with household robotics, but he elaborated that even for self-driving, you have the same problem. "Let's say I train a self-driving car on the streets of San Francisco, and now I want to drive in Detroit or someplace where the streets look very different. It's not going to work because there is a big mismatch. It's just going to crash. So you have these types of issues where there are concerns over where you collected this data and what you are trying to deploy," he said.