Why is it so rare to get real-world data? Too slow to operate, too fake to simulate, and can we automate data collection itself?While remote operations can provide effective physical results, they are slow and have a strong operator bias; simulation generation, while cheap and scalable, cannot guarantee access effectiveness, and perfect crawling in simulation is often a fluke. A natural solution is for candidates to automatically execute and verify crawling on real robots, but this requires the entire collection of looping chips to be sensed, executed, and re-marked. How to build such a complete closed-loop automation system has always been a major engineering challenge pending in the field of robot learning.
Facing this challenge, a team of researchers from Seoul National University and RLWRLD has introduced AutoDex, a real-world grappling data collection system that has revolutionized the data collection paradigm at a pressurized rate of efficiency, from perception to re-automation of all processes.
The AutoDex system, developed jointly by Seoul National University and RLWRLD, is the first fully automated shutdown to capture real-world data collection with ease. The system integrates 20 camera-intensive sensing, collision monitoring movement execution, physical success/fail auto-marking, and four core modules for the weight of the main animal. It successfully collected 3593 crawling trials on 100 diverse objects, and in 500 matching test sessions, AutoDex achieved a 4.8-fold throughput in just 10.3 hours, compared to 49.4 hours of remote operation.
Paper information
Title:AutoDex: An Automated Real-World System for Dexterous Grasping Data Collection
Authors:Mingi Choi, Gunhee Kim, Jisoo Kim, Taeksoo Kim, Taeyun Ha, Jongbin Lim, Hanbyul Joo
Institutions:Seoul National University (SNU), RLWRLD
The code link:https://willi19.github.io/AutoDex/
Readings
Learn the tricks of crawling real-world data that needs to be recorded to crawle the physical results of attempts. This type of data is difficult to obtain on a large scale: remote operations can produce effective physical results, but are slow and with operator bias; simulation-based generation, although inexpensive and scalable, cannot verify access effectiveness. A natural solution is to generate candidate crawling and verify on real hardware, but this can only be achieved during the entire collection cycle, without the need for artificial intervention. This paper proposes AutoDex, an automated real-world data collection system, to fully integrate this cycle: for each candidate generated from replaceable cameras, a system-intensive 20-camera sensor can be used to detect objects in-hand collisions, monitor the execution of data labels between controlled systems, and maintain the physical presence of objects through the use of a database of labels, and to detect the physical presence of objects under controlled systems. Using AutoDex, the research team collected 3,593 crawling experiments covering Allegro and Inspire, covering 100 diverse objects, and simultaneously recorded multipoint observations and robot status logs. In a matching session of 500 tracks, AutoDex took just 10.3 hours, while remote operation took 49.4 hours, achieving a 4.8-fold increase in throughput; the crawling success rate from AutoDex verification database searches reached 76%, while the crawling success rate from simulated verification alone was only 34%.
Demonstration of Effects
Figure 2: AutoDex workstations and their execution examples. On the left: with 6 degrees xArm mechanical arms, multi-camera workstations with interchangeable fingernails and 20 synchronous RGB cameras; the middle row and the right row: each row gives a candidate grappling scheme generated by constraints on walls, shelves, and case scenes respectively, and the corresponding physical execution results; the image is accompanied by an image of the same perspective, and an overlay of the tracked objects in 6D position.
References
From factory assembly lines to home service robots, from in-space maintenance to disaster-site search and rescue, agility is always the holy grail of robotics. Unlike simple parallel claws, agile hands have dozens of degrees of freedom and can interact with objects in multiple ways, such as pinching, holding, holding, holding, holding, etc. This flexibility gives the robot the potential to respond to unstructured environments and unknown objects, but it also brings enormous learning difficulties: strategies must understand complex touch mechanics, finger coordination and force control, not just "close and close".
The rise of deep learning has led to new hopes for crawling agility. From simulation learning to reinforcement learning, from pure simulation training to sim-to-real migration, researchers have tried all sorts of pathways to obtain effective crawling strategies. However, one brutal fact remains unchanged: no matter how realistic the simulation is, physical contact with the real world still exists.
There are mainly two traditional ways of obtaining real physical feedback data: remote operation and simulation verification. Remote operation allows human experts to directly control the execution of crawling by robots, although it is possible to obtain 100% real physical results, but very slowly (an operator may only collect hundreds of tracks a day) and the data inevitably comes with preferences and habit biases of the operator. Simulation verification generates a large number of candidate crawling in the simulation, and then selects high scores of candidates to be executed on real hardware, but the gap between simulation and real makes a large number of candidates ineffective and ineffective in the real world.
The team at AutoDex realized that the real obstacle to large-scale, real data collection was not in grabbing itself, but in grabbing the surrounding 'circular processes': how to automatically find objects, how to safely execute movements, how to judge success or failure, how to prepare for the next experiment. They focused their engineering innovation on this complete automation cycle, which achieved a revolutionary breakthrough in data collection efficiency with systemic design excellence.
Major contributions
The paper proposes AutoDex, the first real-world sensing-executing-marking-repositioning fully closed-loop automation, to provide a large-scale scalable data infrastructure for robot agile operation learning by clutch-level efficiency and physical verification accuracy. Specifically, our main contributions are:
• We built a fully automated data collection loop: AutoDex integrates candidate generation, intensive sensing, collision monitoring execution, physical result marking, and the weight of the main animal into a completely uninterrupted automated system for the first time.
• We achieved 20 cameras of intensive sensing and shadowing positioning: With full coverage of 20 cameras, AutoDex is able to pinpoint objects in extreme conditions of severe hand-held shadowing, solving the most tricky perceptual difficulty of all.
• We designed the crash monitoring safety execution: the system monitors in real time potential collisions between robot movements and the environment, maximizing execution efficiency under the premise of ensuring hardware safety.
• We achieve physical success/fail automatic marking: by automatically judging the physical success of each grabbing of the test by "lift-to-hold" tests, we eliminate the cost and subjectivity of artificial marking.
• We designed the main animal weight placement mechanism: automatically repositioning object postures during trials, actively grabbing candidates under different stability faces of exposed objects, significantly improving data diversity and coverage.
• We achieved 4.8-fold efficiency improvements with a 76% verification success rate: AutoDex reduced its time from 49.4 hours of remote operation to 10.3 hours in a matching collection of 500 tracks; the database with AutoDex Physical Verification captured a success rate of 76%, far above the 34% of simulation verification.
• Our open-source, large-scale data set covering both Allegro and Inspire, 100 diverse objects, a synchronous multi-point observation with 3593 crawling experiments, and a robot status log will be released to the public.
The method
AutoDex's system architecture is composed of five closely collaborative functional modules: a candidate generator interface, 20 cameras of intensive sensing, collision monitoring movement execution, lift-keeping physical marking, and weight placement of the main animal.
The overall workflow is as follows: Once the system is started, the candidate generator (which can be used for any modifiable module, such as a crawling sampler in a simulation or a learning strategy) proposes a candidate crawling gesture. The 20 camera sensing module can locate the target object at any time in the current scenario, even if the robot finger has partially obscured the object. Based on the sensing results, the collision monitor executes the module planning and executes the robot movement, detecting potential collisions in real-time during the approach and contact process and suspending the touch protection.
AutoDex's solution is to deploy 20 cameras around the workstation to "quantitatively overcome" the shadowing challenge. Many fingers wrap objects from multiple directions in the process, creating a severe visual shadowing of the object. It is almost impossible for cameras with only a single viewpoint or a small number of viewpoints to continue tracking objects during the execution of the shot. AutoDex's solution is to deploy 20 cameras around the workstation to "quantitatively overcome" shadowing, forming a near-zero-angle viewfinder.
The collision monitoring movement execution module ensures the safety of the system's long-term autonomous operation. Any accidental robotic-environmental collision in an unattended automation cycle can result in hardware damage or test interruption. AutoDex adopts a layered security strategy: at the movement planning level, it calculates the no-collision trajectory in advance; at the execution level, it monitors the joint force matrix and end executor speed in real time, triggering an immediate shutdown once signs of abnormal shocks or deviation from the designated trajectory are detected. This dual guarantee of "planning prevention + execution monitoring" enables AutoDex to run continuously for decades without the need for artificial intervention.
Raising - maintaining physical marking is a key source of AutoDex data quality. Unlike relying on man-made observation or simulation scores, AutoDex judges crawling success through real physical interaction: robots attempt to lift and hold an object for a period of time after performing a candidate crawling. If an object falls off in the process of lifting or holding, it is marked as a failure; if an object is raised and held steadily, it is marked as successful. This physical marking mechanism ensures that every record in the database is verified in the real world, with unparalleled computerized reliability.
The main animal weighing module is the secret weapon of AutoDex to enhance the data diversity. If the object after each test remains in the same posture, the candidate generator will repeatedly face similar grappling conditions, which greatly reduce the data diversity. AutoDex's replacement module actively moves and reposition objects after each test, systematically exposing objects to grappling under different stable postures. This dynamic exploration strategy can cover a wider range of object-hand interaction configurations in the same number of trials compared to passive waiting.
The results of the experiment
The experimental evaluation of AutoDex revolves around the three core dimensions of data collection efficiency, physical verification effectiveness and data scale.
Collecting Efficiency Pressure Remote Operations: Under the conditions of control variables, the research team compared the time required to collect 500 tracks between AutoDex and artificial remote operations. The results showed that AutoDex took only 10.3 hours, while remote operations took 49.4 hours. AutoDex achieved a 4.8-fold increase in throughput. This efficiency jump is entirely due to the automated shutdown eliminating the time spent on artificial, operational observation and decision making. It is worth noting that these 500 tracks are "match collection", meaning that the set of objects collected by the two models and the candidate sources are consistent, ensuring fairness of comparison.
Overwhelming advantages of physical verification: In order to verify the value of AutoDex physical markup data, the research team compared the actual crawling success rates of three verification strategies: 1) crawling success rates of only 34% from database retrievals of simulated verification; 2) crawling success rates of 76% from database retrievals of AutoDex physical verification. This more than double gap has proven that true physical verification is the most effective way to bridge the real-real-field gap. For downstream crawling strategy learning, using AutoDex verification data means fewer real-world trials, faster strategy convergence and higher final performance.
Using the AutoDex system, the team collected a total of 3,593 crawling experiments covering 100 diverse objects of different shapes, sizes, materials and weights. The data sets contained crawling records from both Allegro and Inspire's mainstream sophistication, with each experiment simultaneously preserving multiple viewpoints from 20 cameras, the state of the robot joints and end-user trajectory, and the sequence of object position estimates, as well as the ultimate physical failure marking.
System robustness: AutoDex demonstrates excellent system robustness during decades of continuous automation operations. The collision monitoring module successfully intercepts multiple potential collisions, actively resets the modules to process the gesture changes of various objects, and senses the modules to maintain the accuracy of position estimates during long-term operation. The reliability of these engineering details is key to allowing AutoDex to move from the "laboratory prototype" to the "practical infrastructure".
Summary & Future of Work
With system-level engineering innovations, AutoDex pushes the entire process of capturing real-world data for the first time to full automation. 20 camera-intensive sensing tackles the problem of positioning under hand-held shadows, collision monitoring and dynamic repositioning ensures system security and data diversity, while 4.8-fold efficiency improvements and 76% physical verification success rates squeeze traditional data collection models with hard data.
Looking ahead, the technical routes and open-source data of AutoDex will generate new research possibilities in many directions. First, the current system is mainly geared towards desktop-level single-object crawling, extending it to multi-object interaction in cluttered scenarios, hand-to-hand synchronization, and agile crawling on mobile platforms, which will greatly expand the application scene. Second, the data collected by AutoDex can not only be used for strategic learning, but can also be used to build more accurate contact physical models, accurate simulator parameters, thus counteracting simulation-real-world migration research.
Readers interested in more experimental results and article details can read the original paper.
This article is for academic sharing only. If there is an infringement, please contact the deleted text.