In the early morning hours of October 30th, April this year, broke robot ((PokeBot) launched the first model product RoboAPI, their world model Poke-Aurora also debuted. Unlike other physical models that released the Demo, broke RoboAPI is a physical API that is geared towards the real physical world, open access, open-box API, and invites the industry at the same time: welcome to try.
As long as the RoboAPI is accessed on the appropriate mechanical arm, the developer does not have to collect data, train the model, go to the command with a prompt, and the mechanical arm can complete tasks that have not been specially trained in a strange environment, verifying the "zero-sample generalization" of the broken model.
Currently, RoboAPI is for non-commercial applications and does not provide products or services to profitable third parties. Developers can access the documentation and developer platform for free through RoboAPI.
One, the RoboAPI
Let the robot model "open the box".
The development of RoboAPI is primarily based on a common industry pain point. Demo videos of robotic models are always surprising to use, but it's something else. A university lab wants to use an open-source model to experience data acquisition, modeling, re-verification of effects, replenishment of data, folding a few rounds, and it's likely to take a month.
One layer is the structural differences: robots do not have uniform formats like text and images, different robots have different structures, sensors and controls, and change anywhere, the model can be "waterproof". The other layer is the generalization itself: even on the same machine, the model often only performs steadily in trained objects and scenarios, replacing unseen environments and objects, the model may lose hands.
Compared to the development of language models, the GPT-3 API has allowed developers not to train and deploy language models on their own to use front-end base models, and RoboAPI wants to do the same in the physical world. Specifically, RoboAPI receives the observation and state of the robot, combined with user instructions, to predict and output the robot's next action almost in real time.
Two, Poke and Aurora
Easy to call zero sample generalization
The first world-breaking motion model supported by RoboAPI is the Poke-Aurora, which has zero-sample generalization capabilities. World-Action Models lay the foundation for the inter-object and cross-scenario migration experience by predicting future images, learning the laws of motion and interaction of objects. However, the ability to model the world does not automatically turn into reliable motion capabilities.
In Poke-Aurora, Breck proposed the "Modulated Residual Coupling (MRC) mechanism" to establish a learning interaction between the two, based on preserving the original computational paths of video and movement respectively. Breck observed that MRC made video prediction more consistent with the direction of learning of motion prediction, allowing models to better understand how objects move and interact and thus better serve motion generation.
In an article published earlier this week, Breaking Bad reveals that they have built a large-scale assessment system that covers nearly 1,000 common objects, nearly 10,000 different instructions, tasks that are divided into moving and organizing objects, aligning gestures and shapes, and interacting with the environment using three types of common actions using tools.
In this assessment, Poke-Aurora had a success rate of 86.0%, 54.1% and 87.7%, respectively, over the GPT6 Astra and the two open-source body models involved in the comparison; neither open-source body model had a success rate of less than 35% on the three types of tasks. More technical details are expected to be published in Poke-Aurora's technical report.
In order to run the process on different robots, Broken optimized the "configuration" when implementing the RoboAPI. Inspired by UMI, Broken defined the "wrist-eye camera + endpoint pose control" configuration: a consumer-grade fish-eye motion camera mounted on the wrist of a mechanical arm, with both observation and control centered on the end of the claw, significantly reducing the threshold and cost of building the configuration.
Three, the "API moment" of physical intelligence.
Let the model be tested by everyone
The real world is far more complicated than evaluation. In the blink of an eye, this is precisely what RoboAPI means most: the generalization ability of a cutting-edge body model should not be determined solely by its creator, but should be verified by any external user at any time.
This is also a response to a long-standing flaw in the body industry. In the body industry, robots and models are provided and used by others. The former is often used interchangeably with the actual capabilities of the models in video demonstration, and the users can only spend a lot of time teaching themselves. RoboAPI hopes to transfer the capabilities of the model directly to the users, allowing it to be tested in different hardware, tasks and environments.
So far, RoboAPI has been tested internally in several university laboratories in the United States and Singapore, and with good feedback, more developers will be able to test it with the official launch.
Autonomous, four times faster
For developers, RoboAPI encapsulates the underlying control of the robot, as few people today care about the control algorithm inside the machine. "The robotic brain" has become the ability to make direct calls, without having to re-research and develop, to focus on real tasks and applications. As more and more people use smart bodies to stream all sorts of tools, software engineers can also use RoboAPI to control robots and build their own workflows.
In mid-October, a U.S. robot model company launched an API test version, in which developers uploaded their own data, modeling them in their hosted algorithms; a few days later, another U.S. startup released a sample of the main API, aimed at clawed-armed robots.
RoboAPI is also an important step forward in the future for breakthrough robots that are committed to building the home robot scene. The founder of breakthrough robots, Xu Zheng, believes that "true generalization capabilities need to be verified more widely in different types of robots, tasks and environments. Robots can only be tested in enough real environments to truly enter the family and face more environments in the future".
The official link:
© THE