Multimodal integration without supervision: visual/IMU/laser radar inter-supervised, without marking

Multimodal integration without supervisionIt's designed to allow vision, IMU and laser radar to pass through.Mutual supervisionThe mechanism of collaborative learning, without artificial marking, is ultimately realized.Unlabeled positioning at centimetersThis direction is at the forefront of the field of autonomous driving and robotics, with the core of complementing the characteristics of different sensors: laser radar provides precise 3D geometry, vision provides rich texture and meaning, and IMU provides high-frequency motion information. Between the three, a variety of surveillance loops can be formed, such as laser radar depth-surveillance depth estimates, or long-term drifting with IMU data that constrains vision/laser mileage, thus relieving the reliance on expensive labeling data.

To achieve this goal depends on the collaborative breakthroughs of three key technologies.Self-supervised/unsupervised learning algorithmsUnVELO is the basis for unlabeled markup, typically represented by geometry and visual loss functions, which can be trained by designing a network without real values; cross-modal contrast learning (such as CrossI2P) can automatically align 2D images with 3D dot cloud characteristics.Multi-sensor integration strategyIn the middle,Close up(e.g. Factor Graph Optimization) constructing various types of observational data as unified optimization factors can significantly improve the accuracy compared to loosely combined.Unmonitored online labelingThe convergence assumption is that existing methods (such as the Double Transformer Framework or DST-Calib) have been able to achieve a calibration error (such as 3.31 cm) and a calibration accuracy greater than 1 cm (0.5 deg) without the need for a calibration board.

Centimeter-level positioning is achieved by relying more on high-precision sensorsClose upWith advanced state estimation methods.Factor graph optimizationFor example, the GIVL-SLAM system can still achieve continuous centimeter-to-minute positioning when the GNSS signal is poor. At the same time, redundancies and complementary mechanisms between the sensors (such as relying on laser radar when light is bad) further enhance the robustness of the system.

In spite of the wide prospects, we are still facingDynamic and degraded environment(e.g. tunnels, moving objects)Computational resources and real-timeandSensors are down.It's a challenge, and it's also a model.Ability to generalizeIn the future, with the continuous optimization of self-supervised learning and compact fusion technologies, stable and reliable non-marking-centimeter positioning will be possible in all scenarios.

Many fans ask the following questions:

Today, let's talk about this sadness together.Almost nine years3D visual technology circles.

Currently, there are nearly 6,300+ active members, mainly involved in the areas of industrial 3D vision, SLAM, autonomous driving, 3D reconstruction, drones, physical intelligence, large models, and technology fronts.

There are many sub-disciplines, including but not limited to: industrial 3D vision, 3D reconstruction, autonomous driving, physical intelligence, large models, widespread models, and technology fronts, including visual competitions, hardware selection, visual product landing experience sharing, academic & job exchange, etc. We also keep up with the latest cutting-edge technological developments and are a hot topic of discussion on our planet.

Summary of the entire series of video tutorials across three planets

Video tutorials are divided intoTwo large platesThe first block is:Core core curriculumWe've thoroughly honed nearly 20 series of video tutorials, content systems, step-by-step, and more importantly, what you've learned about 3D visual technologies like industrial 3D vision, SLAM, 3DGS, autonomous driving and drones, which we'll cover in more detail below.Technology sharing at the forefrontIn a special event, industry leaders and academic leaders were invited to share live content covering top trends such as physical intelligence, autonomous driving, 3D reconstruction, digital people, distributed modeling, 3D generation, and others. This section is not focused on the content and is no longer discussed in detail.

3DGS exclusive video tutorials

3.0.1 3DGS 3D rebuilding series video tutorials

3.0.2 SLAM algorithm video tutorials based on NeRF, 3DGS three dimensional reconstruction

This video tutorial takes you from theoretical and code realization to learning the principles of zero-entry NeRF/Gaussian Based SLAM, reading papers, coding, etc. Theoretical level, from linear algebra to traditional computer graphics, gives you a glimpse into the theoretical foundations and sources of modern 3D reconstruction; the code level teaches you how to reproduce computer graphics, NeRF-related work through multiple hands-on exercises.

3.1 Basic courses

3.1.1 High-precision camera calibration from theoretical to practical combat system instruction

This course mainly consists of two parts: camera labelling and 3D vision; camera labelling includes single-eyed, double-eyed and fish-eyed camera labelling; 3D vision includes stereo vision, structural light and TOF, based on stereo vision.Courses provide corresponding data and code。

Highlights of the CourseIn addition to the basic principles and code that the camera labels, the planet also adds:High-precision camera-labeled improved methods and real-world warfare techniques, which are quite hardcore.。

3.1.2 ROS2 from entry to live video tutorials

ROS2 provides a comprehensive hands-on instructional training on ROS2 from entry-level to real-world video tutorials, introductions and high-level presentations, and systematic learning opportunities for all.

3.1.3 Quad Flyers: Algorithms and Combat

3.1.4 C++ goes from zero-based to advanced

Try to learn through video tutorials, fully understand some of the more difficult modules in C++, master algorithms such as SLAM, learn C++ related job interview skills.

3.2 Industrial 3D visual series video courses

3.2.1 3D reconstruction of video series based on facial structures

3.2.2 Mechanical arm grabbing, 3D point cloud, 3D reconstruction and so on

3.2.3 Linear structural light (single) and three-dimensional system reconstruction tutorials

This course focuses on real-world warfare instruction, hand-in-hand with the implementation of a linear laser 3D scanning system of its own from scratch. Almost every chapter is equipped with corresponding lectures and codes to help you better understand the technology of linear laser 3D reconstruction.

3.3 SLAM series of videos

3.3.1 How to easily pinch LIO-SAM?

3.3.2 Thoroughly dissect the laser-visual-IMU-GPS fusion SLAM algorithm

This video course explains the technical difficulties and frameworks of the SLAM algorithm for laser radar-vision IMU-GPS integration from both theoretical and code implementation and, based on his years of experience in robotics engineering, explains the methods and techniques for multi-module integration in real-world robotic applications.

3.3.3 ORB-SLAM3 theoretical basis + key technical details

3.3.4 Visual-inertial SLAM: VINS-Fusion Principle Precision

The introductory and practical video tutorials for visual-inertial SLAM, combined with the VINS-Fusion source code, systematically align the basic theoretical knowledge of visual-inertial SLAM. The entire course is taught by first-rate algorithm engineers, from basic theory to code analysis, nursery-level teaching, assisting students to grow from small to large.

3.4 Self-driving

3.4.1 Multi-sensor integrated system tutorials for autonomous driving

Mainly divided into two large modules: theoretical and practical, theoretical sections mainly introduce the sensing hardware commonly used in autonomous driving, time synchronization and spatial synchronization between sensors and knowledge of information fusion rationality between multi-sensors; practical, more oriented engineering applications, how to achieve synchronization and integration between sensors in engineering, etc.

3.4.2 Detection of 3D point cloud deep learning targets for autonomous driving

This video tutorial focuses on 3D point cloud deep learning, system analysis and coding of point-based and Voxel-based 3D targeted detection network architectures to help students gain a faster insight and deeper understanding of point cloud depth.

3.4.3 Single-point depth estimation methods: theoretical and real-world video

Video tutorials are mainly divided into two main sections: theoretical and practical, and due to the difficulty of obtaining the depth of the true value of the supervised method, and the effect of the unsupervised method is almost equivalent to that of the supervised method, we focus on the unsupervised method.

3.4.4 Deep learning models in autonomous driving deploy live video

This video tutorial will use a combination of theoretical and practical thinking, first to explain the programming model of TensorRT and the relevant knowledge of GPU/cuda to the extent that it is known; then the course will use three examples of classification, detection and division to demonstrate the detailed programming process and give the relevant code to achieve a truly landing industry share.

Four 3D visual foundations coming in

Considering that many beginners may find it difficult to configure 3D visual software or a visual library, we have prepared the configuration tutorial documentation for each module for you to learn.

5 3D visual source code summary

Below is the source code for the 3D vision industry on the planet:

6 High-Quality Projects Released and Combined