“ A robot's eyeballs turn the spindle itself and can also be part of the perceptual information.
Today's robotics aren't as simple as having two cameras to see the world.
Figure 03 is a special redesign of the visual system, with the camera's shutter speed doubled to the previous generation, and the delay reduced to 1/4; this year's release of the Helix 02, further incorporates head cameras, handheld cameras, touch and body perception into a unified visual movement network.
The more sensors, the more sophisticated the visual system. But when the two eyelids in a robot's head can actually move, track, or even cling together to a target, a long-standing problem arises: how much information is generated when the eyelids move, and can the robot participate in judging the three-dimensional world?
In the latest edition of the robotics journal Science Robotics, a team from the University of York, Canada, published a study on convergent binocular stereo: Depth perception for humanoid robot vision, proposing a convergent binocular stereo algorithm that combines two-dimensional vision (CBS).
It didn't follow the path of stacking data and models. The researchers let the two eyelids, which can rotate independently, cling together to a target, and then used the camera to calculate the depth of field together with the direction, focus points, and the horizontal and vertical differences in the image to the right.
The movement of the eyes also began to become visual information. It is even more remarkable that in situations where double-eyed vision is prone to matching ambiguity, such as in repetitive patterns, this approach outperforms even a series of deep-learning 3D vision methods in paper tests.
01.
My eyes are turning, why isn't the calculation working?
The robot uses two cameras to view the world, and one of its core tasks is to recover the depth of the scene from the difference in the image to the left and the three-dimensional spatial relationship. At this point, it is common practice for humanoid robots to place the two cameras fixed in parallel.
The reason is simple: good luck.
When the right-hand camera remains parallel, the same object's position in the two images occurs mainly in the horizontal direction. The algorithm finds the corresponding points along the same horizontal line, then uses the horizontal line of sight to calculate the distance. The search range is small and the geometric relationship is stable. Therefore, many traditional algorithms, including later deep learning methods of linear vision, follow this set of parallel binary configurations.
But the problem is that people's eyes are never fixed.
When you hold a finger in front of your eyes, both eyes turn inward at the same time, and the two points of view intersect in the same position; the fingers move away, and the two eyes slowly open outward.
Some human robot heads have long been capable of scanning, gathering, and tracking, but these movements are largely confined to where their eyes look. When calculating true depth, the system often returns to a fixed, parallel pair of eyes, or with additional 3D cameras, LiDARs.
So there's a very specific blink of an eye that's been turned around, but deep computing doesn't really use these movements.
The reason for this is not difficult to understand. Once the two cameras rotate inward, the original simple geometry relationship is disrupted. The same point in the left- and right-hand image will not only go wrong, but may go wrong, and the algorithm needs to calculate the horizontal and vertical differences at the same time.
However, this information is not just a computational burden. The paper notes that vertical binary optics themselves contain three-dimensional spatial information related to the geometric properties of the surface, such as depth, size, curvature, tilt. Combined with camera orientation, this information can also be further used for estimates of surface tilt and object posture.
That is, in the past, cameras were fixed down to make double-eyed vision easier to calculate; and once the blink of an eye was allowed to really turn around, the problem, though more complex, had a whole new set of spatial clues.
What CBS wants to do is to get this set of clues really into depth calculation.
02.
How do you turn your eyes in the direction of a point?
The first step for CBS is to make two blinders really look at the same place.
When two cameras simultaneously rotate inward and focus on a target in space, the two axes converge at the same location, which is the focus point. Ideally, this point will fall near the center of the image to the right, with a level and vertical deviation of close to zero.
That's the equivalent of getting the algorithm to the next geometric point.
With this focal point, the robot knows not only where the two images rotate differently, but also how much the two cameras rotate, where they are looking, and where they are looking together.
It will combine geometry and geometry.
Next, CBS looked for the corresponding points in the image from rough to fine, based on the geometry of the camera in the assembly state, while calculating the level and vertical differences, and eventually recovering the depth of the scene.
The paper specifically uses methods such as SIFT characteristics, linear constraints and Gabor waves. The most important thing here is that the whole computation is more of a layer of information that has rarely been used in the past: the camera's own orientation and concentration relationships.
CBS triad matching algorithm
To validate this approach, the research team built a set of two-eyed systems with both cameras moving horizontally and vertically independently, and built a CBS-BM data set.
The data set contains a total of 49 real-life scenes. Each scene, in addition to parallel double-eyed images, also collects 5 to 12 clusters of images from different points of view; it includes both ordinary scenes and specifically includes repetitive patterns, smooth non-characteristic planes and highly self-obscuring objects that make it easy for the three-dimensional matching to go wrong.
It is one of the easiest patterns to look at in a flower that really brings out the advantages of combining two purposes.
CBS-BM data set
03.
Deep learning is more stable where it's easy to spot mistakes.
The paper compares traditional methods such as CBS with SGBM, ELAS, and 8 deep learning methods such as IGEV++, MoCha, RAFT-Stereo, CRE-Stereo.
In general scenarios, long-distance scenarios, and some metrics that require very small errors, some parallel blinking and deep learning methods performed better. But in a combined 49 scenarios, CBS achieved the lowest average depth error and average level of vision error, and the overall ability to compete with the current strongest parallel blinking methods.
The real gap was drawn by the repeating patterns mentioned above.
For example, a row of almost identical objects, or continuously repeating patterns. Although the two cameras take the same scene from different angles, there are too many places in the image that are too similar, the algorithm can easily mistake the number of an object seen by the eye for another similar object in the image.
Once the correspondence is wrong, the gaps and depths behind it will follow.
CBS has more than one layer of information here. In addition to comparing the two images, it also knows where the two cameras are now, where they are focusing on each other. When the images themselves are not clear, the geometry of the group will continue to help the algorithm narrow the matching range.
In the paper's repetitive pattern experiments, multiple parallel approaches showed apparent visual distortion, while CBS was able to use camera direction and focus-point constraints to get results closer to true values.
The gap in the data is also direct. In the repetitive pattern concentration, the average depth of error in CBS is about 0.8 meters lower than the secondary optimization method, and the average visual error is about 100 pixels lower. Considering that these test scenes themselves are no more than 3 meters long, this gap is already quite apparent.
Overall, CBS has a 6.08% pixel depth error greater than 0.40 meters, with a sub-optimal alignment ratio of 39.33%.
This suggests that the robot's own movements can provide additional physical clues when the texture and features in the image are no longer sufficient to give a clear answer.
In addition, CBS has a special advantage: it does not require training for a specific dataset. The paper notes that the performance of the data-driven metric vision method is influenced by the coverage of the training set, while the resulting camera geometry is derived from the actual movements of the robot at the moment.
In other words, how to turn a blind eye is no longer just a mechanical movement, it really begins to take part in robotic judgments of space.
04.
The Future and the Word
CBS has demonstrated how to turn a blind eye into a visual computation, but it has several real-world barriers to actually running on a robot.
One of the most straightforward is speed. Currently, it takes about 69 seconds for CBS to complete a calculation on an AMD Ryzen 7 7700X processor, which clearly does not meet the requirements for real-time sensing by a robot. At the same time, this method is also sensitive to motor accuracy and camera marking, because exactly how much and where the camera turns is itself part of the depth calculation. The paper therefore ranks algorithm optimization, higher precision motors and online marking as a follow-up.
Long-range targets are also a shortboard. The further away the targets are, the smaller the focal points of the two eyelids, and the higher the precision requirements for the motor's fine-tuning. The paper suggests that the future can be combined with confidence estimates, single-eyed depth clues, and other methods to improve the reliability of long-range situations.
And a step further, CBS has not yet solved a question that is closer to the human eye: where should the next eye look? Human vision is not mechanically stuck in a point, but is constantly shifting focus according to tasks. The paper also lists focusing on strategy, central lensing, and multi-resolution perception as a future worth exploring.
Therefore, CBS is more like adding an underlying capability to active vision.
In the past, robots have shifted their eyelids more by changing the direction of vision; this paper further demonstrates that the movement of the eye itself can also be part of perceptual information.
From seeing a mouse, to really knowing where to look, how to look, and how to use the movement of the eye to understand space, the human-robot vision may still need to move forward.