Robots pick up household chores by observing people

It initially failed, missing the handle entirely at times, grabbing it in the wrong place, or pulling it incorrectly. However, after several hours of practice, the robot was able to open the door.

"Imitation is a great way to learn," said Bahl, a Ph.D. student at the Robotics Institute (RI) in Carnegie Mellon University's School of Computer Science. "Having robots actually learn from directly watching humans remains an unsolved problem in the field, but this work takes a significant step in enabling that ability."

Robots pick up household chores by observing people.

In-the-Wild Human Imitating Robot Learning, or WHIRL for short, is a new technique that Bahl created with Deepak Pathak and Abhinav Gupta, both professors at the RI. The one-shot visual imitation algorithm WHIRL is effective. Robots are well suited to learning household chores because they can directly learn from videos of human-human interaction and generalize that information to new tasks. In their homes, people constantly carry out a variety of tasks. A robot can observe those tasks with WHIRL and collect the video data it requires to eventually figure out how to finish the job itself.

The team modified an off-the-shelf robot by adding a camera and their software, and the robot quickly learned to perform more than 20 different tasks, such as opening and closing cabinets, drawers, and appliances as well as placing lids on pots, pushing in chairs, and even removing trash bags from bins. Each time, the robot observed a human perform the task once before starting to practice and learn how to complete it independently. The group presented their research at the Robotics: Science and Systems conference in New York earlier this month.

"This work presents a way to bring robots into the home," said Pathak, an assistant professor in the RI and a member of the team. "Instead of waiting for robots to be programmed or trained to successfully complete different tasks before deploying them into people's homes, this technology allows us to deploy the robots and have them learn how to complete tasks, all the while adapting to their environments and improving solely by watching."

The majority of current techniques for teaching a robot a task rely on reinforcement learning or imitation. In imitation learning, a robot is taught to perform a task by being manually operated by humans. Before the robot learns, this process must be repeated several times for a single task. In reinforcement learning, the robot is frequently instructed to adapt its training to the real world after being trained on millions of examples in simulation.

When teaching a robot a single task in a structured environment, both learning models perform well, but they are challenging to scale and implement. Any video of a person performing a task can be used by WHIRL to learn. It is easily scalable, not restricted to a single task, and capable of functioning in actual home environments. Even now, the team is working on a WHIRL variant that learns from watching videos of people interacting on YouTube and Flickr.

The work was made possible by advancements in computer vision. Computers can now comprehend and model movement in three dimensions using models trained on internet data. The team trained WHIRL by using these models to comprehend human movement.

A robot can carry out tasks in its environment by using WHIRL. The chairs, garbage bag, doors, drawers, lids, and other objects were not altered or moved to accommodate the robot. The robot failed its first few attempts at a task, but after a few successes, it quickly grasped how to complete it and mastered it. While a robot might not perform the task using human-like movements, that is not the intended outcome. Robots and people move differently and have different parts. The fact that the outcome is the same is what counts. A door is unlocked. The switch has been disabled. The water supply is running.

According to Pathak, for robotics to be scaled in the real world, the data must be dependable and stable, and the robots must improve independently as they practice in their surroundings.

Reference:Carnegie Mellon University. "Robots learn household tasks by watching humans: Novel method allows robots to learn in the wild."