The first author of the DexNDM achievement is Liu Xueyi, a Ph.D. student at Tsinghua University's Cross-Cut Information Research Institute. The corresponding author is Li Yi, an assistant professor at Tsinghua University's Cross-Cut Information Resea

When can robots with dexterous hands help humans in screwing or cooking at home, just like those in factories? To realize this vision, DexNDM is born to solve the sim-to-real challenge of dexterous manipulation skills.

Paper Title: DexNDM: Closing the Reality Gap for Dexterous In-Hand Rotation via Joint-Wise Neural Dynamics ModelnPaper Link: https://arxiv.org/abs/2510.08556nProject Website: https://meowuu7.github.io/DexNDM/nYouTube Video: https://www.youtube.com/wa

Background - High-Agility Complex Tool Teleoperation

Performing complex tool teleoperation with high dexterity, such as controlling a robotic hand to use a screwdriver or hammer, is a longstanding core challenge in the field of robotics. Traditional direct mapping teleoperation schemes, which involve d

To overcome this bottleneck, we propose a semi-autonomous teleoperation paradigm. The core idea is to decompose complex teleoperation tasks into a series of autonomous and stable atomic skills that robots can execute. Operators only need to provide h

Among many atomic skills, in-hand object rotation is a crucial and challenging basic ability. It is not only a concentrated expression of the dexterity of the dexterous hand, but also a prerequisite for using most tools. However, in-hand rotation inv

To this end, we propose DexNDM, a new method aimed at learning general and stable atomic skills. DexNDM is designed to overcome the limitations of existing work, enabling the dexterous hand to master the skill of stably rotating various objects along

Based on the powerful and stable rotating atomic skills provided by DexNDM, we have finally built a highly flexible and robust semi-autonomous teleoperation system. In this system, operators can easily guide the dexterous hand to complete previously

Unparalleled Flexibility

Highlight 1: Full-scene object coverage: from micro to ultra-long, from simple to complex, all can be precisely controlled

First Time Overcoming Continuous Rotation Under Extreme Challenges

DexNDM breaks through the ceiling of existing intra-hand rotation technology and for the first time realizes continuous and stable rotation of long objects (such as sticks and pens) along the long axis under extremely challenging wrist postures such

Figure 1: Rotating small objects and long objects

Wide object coverage, perfect for complex geometry

Wide Object Coverage

DexNDM has made significant breakthroughs in the diversity of operable objects, achieving full coverage from tiny to slender and from simple to complex geometries, with a breadth far exceeding any previous work (as shown in Figure 2).

To quantify this advantage, we directly compared it to previous work, Visual Dexterity, which performed best in rotating complex geometries. The results show that even though DexNDM uses a smaller and more versatile Leap Hand than the customized D'Cl

Furthermore, DexNDM has created an unprecedented capability. For the first time, we demonstrated the use of universal manipulators such as Allegro and Leap Hand to stably rotate complex geometries with uneven surfaces in challenging wrist postures su

Rotating Complex Geometry

Highlight 2: Full attitude precise control: Any wrist posture and various rotational axes can be moved freely

In addition to its excellent versatility in object types, DexNDM is also characterized by its strong adaptability to wrist posture and rotational axis. Regardless of the orientation of the manipulator or the axis along which the task requires the obj

Diverse wrist orientations

【Highlight 3】A highly dexterous and robust tele-operation system with flexible hands, capable of handling various tools and long-range assembly tasks

We have taken DexNDM's powerful intra-hand rotational capability as an 'atomic skill' to build a teleoperation system with far greater flexibility than traditional solutions. In this system, operators simply control the pose of the robotic arm throug

Furthermore, the super robustness of DexNDM strategy enables teleoperation systems to be competent for long-horizon assembly tasks that require extremely high stability. For example, we successfully completed the complete assembly process of installi

Fig. 5: High dexterity and high robustness dexterous hand teleoperation

Technical Breakthrough

DexNDM's powerful object rotation capability in the real world is achieved through its innovation in the sim-to-real approach (Figure 6).

Specifically, the core of this approach is a joint-level dynamic model, which effectively fits a small amount of real-world collected data, and adjusts the actions of the simulation strategy accordingly, thereby bridging the dynamic deviation between

The author adopted a fully automated data collection strategy to collect diverse real-world interaction data with minimal human intervention. The joint-by-joint dynamic modeling and fully automated data collection strategy proposed by DexNDM provide

Based on the joint dynamics model of a dexterous hand trained in the real world, the author trains a residual policy network on the original policy to output a correction term according to the original policy network's command output, aiming to compe

Fig. 6: Overview of Methods

Joint-level Neural Dynamic Model

Unlike the more direct modeling of whole hand and object interaction dynamics, the joint-level neural dynamic model decomposes the complex interaction dynamics at each joint, predicting the state of each joint independently at the next moment from it

The author has verified the three key properties of the joint-level neurodynamic model through theoretical analysis and experiments, namely high expressiveness, high data efficiency, and strong generalization ability. This kind of generalization abil

Automated Data Collection System

Based on four principles: (i) the collected data is related to the transfer distribution of the strategy network, (ii) there is an object load, (iii) the distribution covers a wide range, and (iv) it is easy to expand, the author has built an automat

The implementation method is simple: place the robot hand in a container filled with soft balls. Then the author replays the actions from the simulation-based strategy in an open-loop manner, which provides a coarse-grained distribution prior (i). Th

Advantages of Joint-level Dynamic Modeling under Biased Data Distribution

Training Based on Residual Strategy

Conclusion

DexNDM has taken a solid and significant step forward in solving the core problem of robotics, Sim-to-Real. It addresses the challenge of learning reliable real-world dynamic models for dexterous manipulation and achieves unprecedented dexterous mani

Although DexNDM has its limitations, the author believes that it is just a beginning. As the crown of humanoid robots, dexterous hands are destined for a bright future and will surely shine.