Have you ever fantasized about having a robot sidekick? How about a machine for all your daily, time-consuming household chores, so you can sit back and relax? That could well be in our future if a team of researchers headed by MIT and the University of Toronto has anything to do with it.
The researchers have developed a virtual system that can teach robots to perform detailed household chores, potentially turning that fantasy into reality.
The full paper is available here.
The system
Inspired by the Sims video game, the system, called VirtualHome, is a 3D house environment capable of generating videos and simulating different household tasks that can be performed by a virtual “agent.”
The researchers used nearly 3,000 programs of various tasks to train the system. These tasks are broken down into specific subtasks for the virtual agents to understand.
For example, a task like “making coffee” would also include the step “grabbing a cup.”
“Describing actions as computer programs gives clear, unambiguous descriptions of all the steps needed to complete a task,” said Xavier Puig, a doctoral student in the Computer Science and Artificial Intelligence Laboratory (CSAIL) at MIT.
“We use these programs to instruct virtual characters to execute tasks in a simulated household environment and show how these tasks can be reconstructed from either written descriptions or even video demonstrations.”
The virtual agent can complete 1,000 of these actions throughout eight different scenes, including a living room, kitchen, dining room, bedroom, and home office.
Why virtual?
In order to complete tasks, robots need explicit, clearly defined instructions — they can’t just infer and reason like humans.
For instance, if you tell a human to “switch on the TV and watch it from the sofa,” they can infer that they have to grab the remote control, and then physically sit or lie on the sofa. A robot however, would need those latter two instructions.
So, the researchers developed VirtualHome as a way to compile a large database of household tasks described with the detailed language that robots require.
To accomplish this, the team first collected verbal descriptions of household tasks and translated them into simple code. For instance, the directions for the previous example would be coded as: walk to the TV, switch on the TV, walk to the sofa, sit on the sofa, and watch television.
Once these programs were developed, the team put them through the VirtualHome 3D simulator and translated them into videos. The virtual agent would then watch and execute the tasks described by the programs.
“Our model successfully demonstrated that the agents could learn to reconstruct a program, and therefore perform a task, given either a description: ‘pour milk into glass’ or a video demonstration of the activity,” said Puig.
Next step: Teaching robots
Since VirtualHome includes a large database of detailed instructions and videos that can teach a virtual agent, the team now hopes to use this information to teach robots how to perform the tasks.
“Eventually, we hope that a robot would watch these videos, and by using a positive-feedback learning system, could recreate the task in a person’s home,” said Puig.
Furthermore, the team wants to someday train robots using actual videos instead of the simulations.
“One next step is to develop VirtualHome so that it can train robots with actual live-action videos, rather than the animated simulation videos that we used,” Puig said.
“This would allow the system to teach robots to do things just by having them watch a video somebody posted on YouTube.”
Ultimately, the researchers believe, VirtualHome will someday teach robots to assist us inside our home.
“You could imagine a robot being able to anticipate what will happen in a home and acting as an assistive technology by anticipating wants and needs, especially for the elderly,” said Puig.
