Computer scientists from the University of Bonn in Germany have developed a self-learning computer program that can look several minutes into the future.   

This development will enable robots to anticipate the actions of humans, allowing machines to work side by side with people.

“If the action a robot should do depends on the action of a human, the robot has to anticipate the future actions of the human,” said Jürgen Gall, a professor in the Institute of Computer Science at Bonn and lead researcher of the study.

Using this software, robots can provide humans with necessary parts, tools or ingredients at the right time.   

“Nobody wants to wait for a robot,” said Gall.

Gall and his team will present their paper at the 2018 annual Conference on Computer Vision and Pattern Recognition, June 19-21, in Salt Lake City.

Working with humans

The ultimate goal of this ongoing study is to teach robots to predict the timing and span of activities, minutes or hours before they happen.

This development could easily apply to robots that clean and work in the kitchen.

A kitchen robot could pass ingredients to a cook as soon as they are needed, preheat the oven, and let a cook know if he or she missed a step.

A vacuum-cleaning robot could know to avoid the kitchen around dinner time and move into a less busy room.  

“Robots have to anticipate the future actions of the humans, otherwise they will not able to do the right action at the right time, and human-robot collaborations remain frustrating for the humans,” said Gall.

Training the computer program

To train the program, the researchers showed 40 six-minute-long videos of people preparing different salads, which the program “watched” for four hours.

In each video, there were about 20 different actions with precise details about the starting time and duration of every action.

From observing the videos, the program learned the order in which different chefs add their ingredients. The program can then use this knowledge to predict future actions in new, similar situations.

“If the network is trained, it anticipates the future actions in previously unseen videos,” said Gall.

Testing the program

To test the effectiveness of the learning process, the researchers showed the program videos it had never seen before.

The researchers informed the program what happened in the first 20 or 30 percent of one of the new videos and asked it to predict what would happen in the rest of the video.

“Accuracy was over 40 percent for short forecast periods, but then dropped the more the algorithm had to look into the future,” Gall said in a statement.

When the program was asked to predict activities that occurred more than three minutes in the future, the accuracy rate dropped to 15 percent.

However, the researchers did not let the program off easily. They only considered it correct if it predicted both the activity and timing.

While this success rate may not yet seem impressive, it serves as the basis for technology that could soon have greater intuition than humans.

What’s next?

The researchers want to make it clear that this study is only the first step into a new field.

They are working on improving the program’s ability to predict what happens in the first quarter of a video, because it performed much worse if the researchers didn’t give it a hint.

They also wish to limit the annotations used in each explanatory video.

“We want that the approach directly learns from videos without additional annotations,” said Gall.

“While we have already achieved a major progress in this direction, there is still a performance gap compared to a network that is trained on videos where the start and endpoint of each action are annotated.”

Additionally, the researchers recognize that the future is not entirely deterministic and holds multiple potential outcomes.  

In an effort to make the machine as knowledgeable as possible, the researchers are working to expand their variety of training videos.

Altogether, there is still a lot of ground to cover, but this research serves as a stepping stone for developing a new type of robot that can work alongside humans without having to be told what to do.

Maybe someday soon a robot helper could wake you up with eggs and bacon on Saturday morning, without you having to program it to.