Robot grips intuitively and moves objects like a human

Agile Justin has sensors in its two, four-fingered robot hands. It uses them to 'feel' objects. The robot recently demonstrated its skills at the automatica trade fair. Robotics researchers consider dexterity to be a major challenge. "Using new methods of learning artificial intelligence, we have succeeded for the first time in demonstrating abilities with the highly developed mechatronic DLR hand that come close to human dexterity. In particular, rotating and manipulating objects without camera inputs and with an hand pointing downward was a challenge," says Berthold Bäuml from the DLR Institute of Robotics and Mechatronics in Oberpfaffenhofen. Bäuml's AI research has been advancing the development of the humanoid robot Justin for many years. Justin catches balls thrown to him, for example, or recognises materials by sensing them with its fingers. Now, learning AI has significantly expanded Justin's capabilities.

"Turning the object is good; dropping it is bad"

The researchers used what is referred to as deep reinforcement learning. This means that Justin learns completely on its own, only by being told what is correct or incorrect. In this application, for example, it was instructed that 'turning the object towards the target is good', while 'dropping the object is bad'. Machine learning of skills takes only two to three hours in a simulation; then Justin masters the task. It develops the movement strategy for the fingers in a similar way to a human, through trial and error. "The robot can then directly execute its solutions in the real environment. Even we as researchers are often surprised. The possibilities of our new learning AI methods are fascinating," says Berthold Bäuml.

Thousands of examples for tens of thousands of things

The researchers also achieved another first in AI robotics. Justin is able to grasp arbitrary objects with its multi-fingered hands. For this, the researchers solved two major problems using AI. Firstly, using 50,000 examples, the robot learned what is referred to as a 'foundational model', which allows it to infer the shape of the back of an object from the visible front. Justin has thus acquired an understanding of three-dimensional shapes. Secondly, Justin can then find an optimal position for its hand and adjust its twelve finger joints appropriately. Humans grasp objects intuitively. In order to convey this intuition to the robot, another AI system was fed with thousands of examples for tens of thousands of objects. "The computations took a few days. But now the robot can recall the correct grip within fractions of a second," explains Bäuml. At automatica, Justin demonstrated its dexterity with objects handed to him by the visitors to the exhibition.

"The work on AI-based dexterous manipulation represents a worldwide scientific breakthrough. Never before have hands been able to manipulate such a large variety of objects so quickly and so precisely. This success was only possible through the combined use of the latest AI methods with model-based approaches and many years of robotics experience," says Alin Albu-Schäffer, Director of the DLR Institute of Robotics and Mechatronics. "Until now, it was often assumed that you needed a very large amount of data to learn dexterous object handling with robotic hands. In contrast to data for large language models, experimental data in robotics, even if generated by simulation, is very costly. The DLR research team has found a way to require less data by using more robotics knowledge and a new learning architecture. Using their new approach, the team can even train these models on a single computer and does not require a large computing cluster."

Possibilities for industrial production and healthcare

The next big step in robotics could be in industrial manufacturing. Multi-fingered robotic hands could assemble very complex and small parts if they master flexible, standardised intelligent manipulation. DLR researchers are working on this in the 'SmartHand' project, for example.

Robotic assistants are also an important topic in healthcare. In the future, they could assist caregivers and give people with reduced abilities back a higher degree of independence in everyday life. The intelligent manipulation skills developed at DLR are also fundamental to this. These skills apply, for example, to the safe grasping cups, bowls, boxes or other diversely shaped household objects.