The human eye never goes to the end of a dead-end procedure when observing a strange thing.
Read a handwritten handwritten letter, the most difficult to recognize are often the first few words. Seeing more, we gradually understand the pen and ink writing habits of the correspondent, and the later words become more familiar.
This seems like a natural thing to do, but it's not easy to implement for visual AI.
After a picture enters the model, which areas need to be focused on, how the characteristics of the different locations are associated, the model is already able to make adjustments based on the image in front of it. In many existing architectures, the rules for determining how information is learned, how much to update, and what old information should be retained are still largely determined by the training phase.
A research team from the MIT and the Institute of Automation of the Chinese Academy of Sciences is trying to push this step forward. Together, the two have proposed VisionHOPE, the first universal visual backbone network to be built at the core of a self-modified learning system.
It allows the model to read and save image information while simultaneously updating its learning method based on what it has read. The areas seen before can participate in determining how the traits behind it are linked, how much learning progress should be made, and how much memory should be retained.
VisionHOPE-S, which has a Top-1 accuracy of 85.2% on the ImageNet-1K image classification benchmark, has a top-1 accuracy of less than 60% on VMamba-B (89M) and an accuracy of 1.3 percentage points; VisionHOPE-B has an accuracy of 85.6%. At the same time, the core computation of its automated modification mechanism has increased linearly with Token.
After its release, Hugging Face was ranked No. 2 on the Daily Paper List, and the GitHub open-source repository garnered nearly 1000 stars in a short period of time. Currently, the research team has made all code, underlying algorithms and pre-training weights available to the community under the MIT protocol.
Title: VisionHOPE: Visual Backbones as Self-Modifying Learning Systems
Hugging Face paper page:
This is the first episode of the first season of Hugging Face VisionHOPE.
GitHub code:https://github.com/PSRben/VisionHOPE
The visual model is becoming more and more eye-catching.
But learning rules can also be more flexible.
To understand what VisionHOPE is all about, let's look at the changes in the visual spine over the years.
The early convolutional neural network (CNN) was good at capturing information from local edges, textures, etc. It used well-trained convolutional core extraction characteristics that required multiple layers of computation to establish connectivity in the image. The later visual Transformer (ViT) allowed different locations to interact directly based on the current image. The model saw a larger range of content, but the projection parameters for generating attention remained fixed, and the computation of total attention increased with the number of tokens squared, the larger the size of the image, and the greater the computational pressure.
Later, state-space models such as Mamba reduce computational complexity to linearity by passing states along the sequence. State transfers can change with the current input, but are still determined by the training phase responsible for the mapping of the input-generated state transfers. TTT (test-to-train) is another step forward: it sets up a small learner within the network that allows the learner to process images to update its own parameters. However, the rules for generating learning expressions, as well as controlling the update step length and memory retention, are still mainly set during training.
Along this path, researchers have been expanding the model's range of adaptations to the current cube: from local characteristics, to global interaction, to dynamic states, and to parametric updates of internal learners. VisionHOPE further incorporates the parameters that determine how to learn into the image's internal updating process.
Figure 1 shows different visual architectures adapting to current images: from local characteristics, global interaction and state transfers, to internal learner updates, and to self-modification learning. The arrows indicate a gradual expansion of the range of adaptation within images.
The design builds on the self-referencing mechanisms of nested learning (Nested Learning, NL) and its HOPE architecture. Simply put, the model processes the content that accumulates while imaging, not only for the final identification results to be generated, but also to be involved in the adjustment of the learning rules for the next step.
This is also relevant to the position of the visual spine itself. Whether it is image recognition or multi-mode systems that need to combine images with language, the visual spine is responsible for providing the most basic visual representation. Previously, MMVP has revealed flaws in the basic visual recognition capability of the multi-mode model; research by Cambrian-1 suggests that improving visual representation can further enhance the multi-mode capability.
Table 1 How different visual architectures adapt to current images
It's a great way to learn how to cook and remember:
VisionHOPE's three key designs
Allowing the model to modify its own learning rules can expand the model's expressiveness, but also creates new theoretical and engineering problems. In building the general visual backbone of SRNL using self-referential embedded learning modules, the research team focused on three issues: how memory is organized and updated, how self-referential feedback stays stable, and how sequential iteration adapts to two-dimensional images and efficient computation.
The five types of memory collaboration can help you learn how to play and change the picture.
The core of VisionHOPE is a module called SRNL (which refers to embedded learning). It can be understood as a set of memory systems that are constantly adjusted as the graphics process is read.
Among them, content memory stores image information for subsequent reading and output; key memory is responsible for generating write-in indexes that associate new information with existing memory; value memory is involved in generating internal learning goals that determine the direction of current learning to be modified; learning rate memory controls the extent of each update; and retention rate memory determines how much old information is retained when updated.
Partition of five types of memory
Each time a new image enters the model, it starts from the initial state of training secondary education. As the new image characteristics enter the SRNL one by one, the existing memory master becomes the corresponding Key, Value, learning progression and retention rate. The five memory classes then construct the learning goals according to their own state and compare the goals to the current response, receiving their own learning signals and completing updates.
Returning to the example of handwriting: after identifying the previous few words, we get not only the content of these words, but also a little new knowledge of the writer's handwriting. In retrospect, which pencils need special attention, which similar writing methods can be quickly classified, and judgments may change. SRNL hopes to make similar continuous adjustments within the model, but it changes the specific rules of computation such as character correlation, learning steps and memory retention.
Such updates also need to avoid having too much memory. In repetitive textures of skies, grasslands or buildings, adjacent areas often contain highly similar information. If new features are added each time, repetitive content may accumulate. VisionHOPE uses Delta gradient decreases (DGD): check the response of the current memory to the Key before writing, then write down the difference value that needs to be corrected, reducing ineffective repetitive accumulation.
The key is that these changes occur during the processing of the same image. When the model reads the back area, it uses the memory that the front area has already participated in, without having to wait for the image to be processed and then retraining the model again.
Learning rules can change, but they must remain stable
Having memory involved in the adjustment of its subsequent updates carries a direct risk: feedback may be amplified repeatedly. If one step of the update increases the memory value, the next step of the learning signal may also be stronger, which then drives the memory further. This cycle may ultimately cause the training to lose stability.
For this purpose, the team designed two layers of stability constraints.
The first layer is the soft injection ceiling, which smoothly limits the injection intensity of each new feedback. The higher the retention ratio of old memory, the less new feedback is allowed to be injected, so that both are kept within a stable range;
The second layer is the spectral interruption constraint, which limits the amount of memory already placed during the transmission process, and is only effective when the candidate's step length exceeds the stable upper limit.
These two layers of constraint have theoretical support. The paper demonstrates that, under the joint action of the two, the spectrum of the five types of memory will not grow beyond the initial state along the scanning path, either by token forwarding or by block forwarding. The model is able to continuously modify the internal learning method while amplifying feedback to the extent that it can be demonstrated.
A picture is up and down, and calculations can't just go down one path.
Another problem that needs to be solved is computational efficiency. The SRNL itself moves along the sequence, but the image is two-dimensional: it relies solely on a single direction, and it is difficult to make full use of up-and-down spatial relationships; if all tokens are strictly updated, a series of bottlenecks will form.
VisionHOPE uses four-dimensional scanning: along lines, columns read images straight and reverse, forming four independent paths. Each direction has its own five types of memory; at the end of the scan, the results are restored to two-dimensional space and weighted together by a channel, allowing different locations to gather information from four directions.
In computational efficiency, in order to accelerate further, the model considers an entire row or a whole row as a computing block. Each location in the block is based on the memory state of the starting point of the block, simultaneously generating the intermediate amount of the update needed; memory updates are then accumulated in sequence in the block and passed on to the next row or next row.
The actual test showed that in the single-scale model P-VisionHOPE-S, 14 tokens were set as a block, and the reasoning speed reached 2.86 times that of each version of the token; the Top-1 accuracy of the two was 82.3% and 82.4%, respectively, with a gap of only 0.1 percentage points.
Image 2 -- VisionHOPE structure. On the left is a two-dimensional algorithm running along a row of two-way scans that independently maintains five types of memory in each direction, maintains stability of updates through step controls, and converges as a gateway after spatial restoration; on the right is a complete VisionHOPE Block that combines local curvature, front-end network and residual connectivity.
Let the experiment speak:
Accuracy, spatial understanding and computational efficiency
Whether or not a new learning mechanism is worthwhile will ultimately return to the task of image recognition itself.
The team first tested image classification capabilities on ImageNet-1K. At 224×224 input resolution, VisionHOPE provides three sizes of Tiny, Small, Base. The results of the comparison with ConvNeXt, VMamba, and the paper also provides a complete comparison of the ViT and TTT skeletons.
Table 3 with the results of the classification with ConvNeXt, VMamba on ImageNet-1K
On a smaller scale, VisionHOPE-T has less than 3.4M parameters compared to VMamba-T, with an accuracy of 1.5 percentage points. VisionHOPE-S has parameters less than 60% of VMamba-B and an accuracy of 1.3 percentage points.
Of course, identifying which category an image belongs to is the basic capability of a visual model. Understanding real images also determines where the object is, how the outline is divided, and what the different areas of the image belong to. For this, the team also accessed VisionHOPE's downstream task: using Mask R-CNN to test COCO target detection and instance division, and using UPerNet to test ADE20K semantic segmentation.
Table 4 Performance on the COCO test split and the ADE20K semantic split
Compared to COCO's 1× training setup and ADE20K, all three sizes of VisionHOPE achieve the same scale optimum or parallel optimum. Performance improvements extend from image classification to tasks requiring spatial positioning and boundary judgement, illustrating the potential of this mechanism as a general visual backbone.
In addition to the figures on the chart, the research team also tried to observe how the model handled the images.
In the effective sensing field contrast of the second network block, ResNet-50's response is mostly localized; DeiT-S is closer to the center than TTT-based ViT3-S; Vim-S has a clear direction. P-VisionHOPE-S covers a larger two-dimensional region while maintaining a clear central response. This means that it can integrate a larger range of image information at a shallower level.
Figure 3 shows the effective perceptual field contrast of different visual skeletons in the second network block. The deeper the input is 512 x 512, the deeper the color indicates the greater the relative influence of the region on the central characteristics.
Another set of visualizations shows the spatial distribution of learning steps in images: larger learning steps are often associated with the subject object; significant structural overlap, background and relatively flat areas are smaller, and intuitively show that it does indeed adjust the learning intensity internally according to the image content.
Figure 4 shows the spatial distribution of the image content with the actual learning steps. Each group of behaviour input images, and the following behaviour learning steps; dark color indicates that a relatively larger step length is used within the image.
Larger regions are more closely related to the main objects in the diagram and significant structural overlap, with smaller backgrounds and flat regions. This indicates that the intensity of internal updating changes with the spatial content of the image, and that the image content is also involved in determining how the model updates memory at different locations while being extracted. It should be noted that step size does not equal the actual change in parameters, which also depends on the size of the learning signal.
The code is open.
It can be used directly or broken down into modules.
The results of VisionHOPE's research are now available as open source. The team has opened up training and evaluation processes, underlying algorithms, and pre-training weights, allowing developers to directly load the entire visual backbone to perform classification or downstream tasks, or to take only a portion of the modules to try to access their existing networks.
Specifically, the project offers four call levels:
These four interfaces cover a wide range of needs, from basic sequence learning studies, to two-dimensional visualization algorithms, complete network modules, and pre-trained model applications. SRNL, VisionHOPEOperator, and VisionHOPEBlock both maintain the shape of input and output voltages, and support PyTorch automated navigation to integrate existing models.
In terms of deployment, the project provides installation examples for Linux, NVIDIA GPUs, CUDA 12.1 and Python 3.10. The underlying CUDA algorithm is automatically compiled on the first call. Pre-training weights can be downloaded from Hugging Face, through `create_model` loading.
From reading a graph,
To gain more visual experience
What VisionHOPE is now able to do is allow the model to update the content memory while processing the same image, as well as the rules that affect subsequent learning. The synchronization of the five types of memory, the constraints of stability in the self-modification process, and quadratic scanning and block computation, together make this a trainable, computable, verifiable path to realization.
It also leaves new research space.
For example, in medical imaging, remote sensing and industrial detection, tiny pathogens and rare defects are often buried in a large amount of visual information. Allowing models to adjust the associated features and learning steps based on the image content may help to capture these details, but the actual effect also requires validation of the corresponding task.
The longer-term direction is to extend the adaptability of this image to continuous observation. Long video understanding, autonomous driving, and cognitive intelligence all involve the connection of the front and back images, and researchers can try to have the five types of combined memory cross the continuous loop, enabling the information seen in the past to be involved in the follow-up processing. However, the current VisionHOPE processes each new image and the memory will still be restored to the initial state trained, and there is no continuous learning across the loop. To make the memory truly accumulate over time, it is also necessary to solve the problem of when to restart the scene switching, and how to avoid the continuous propagation of false information.
In addition, SRNL serves as a stability-guaranteed sequential self-modification module, as well as a context-based memory for language models, as well as a cross-modal memory that allows images and text to participate in updates, providing a technical input for further research.
The handwritten handwriting of the handwriting is easier to read as the reader recognizes the content while forming an awareness of the handwriting. VisionHOPE accomplishes a similar step in the visual spine: let's see what we've seen and be really involved in deciding how to learn next.
From self-modification within a single image to a visual system capable of preserving experience across images, there are many technical questions to be answered. But now, researchers have been able to start with open-source models, calculators, and codes and continue to test this route to see how far they can go.
Installation and call examples
In Linux environments with NVIDIA GPUs, CUDA 12.1 toolchain and Python 3.10, the installation can be completed by following the following commands:
conda create -n visionhope-demo python=3.10 -yconda activate visionhope-demogit clonecd VisionHOPEpython -m pip install torch==2.1.0 torchvision==0.16.0 \--index-urlpython -m pip install -r requirements/classification.txtpython -m pip install --no-deps -e .
If you want to use the pre-training model directly, you can download the weight from Hugging Face and upload it with the create_model. The warehouse also provides the prepar_for_inference function, which can be used to accelerate the reasoning of the original merge model parameters after the load; if you want to continue training, you should keep one unmerged model:
python -m pip install --upgrade huggingface_hubhf download PSRben/VisionHOPE visionhope_tiny.pth --local-dir weights
import torchfrom visionhope.models import create_modelfrom visionhope.inference.prepare import prepare_for_inferencemodel = create_model("visionhope_tiny",checkpoint_path="weights/visionhope_tiny.pth",).cuda().eval()prepare_for_inference(model)with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):images = torch.randn(1, 3, 224, 224, device="cuda")logits = model(images)print(logits.shape) # torch.Size([1, 1000])
If you focus on architectural exploration, SRNL, VisionHOPEOperator and VisionHOPEBlock both keep the input and output voltage shape unchanged, and support PyTorch automatic navigation, which can be embedded in existing models like standard network layers:
import torchfrom torch import nnfrom visionhope.models import SRNL, VisionHOPEOperator, VisionHOPEBlock# 1. 调用独立序列模块处理 [B, N, D] 特征序列srnl = SRNL(dim=128, head_dim=16, chunk_size=64).cuda()x = torch.randn(2, 257, 128, device="cuda", requires_grad=True)y = srnl(x)y.square().mean().backward()# 2. 调用二维算子直接处理 [B, C, H, W] 矩形特征图operator = VisionHOPEOperator(dim=64, head_dim=16).cuda()features = torch.randn(2, 64, 14, 20, device="cuda")out = operator(features)# 3. 用完整 VisionHOPEBlock 组合出一个轻量级分类网络net = nn.Sequential(nn.Conv2d(3, 64, kernel_size=4, stride=4),VisionHOPEBlock(dim=64, mixer_dim=32, head_dim=16),VisionHOPEBlock(dim=64, mixer_dim=32, head_dim=16),nn.AdaptiveAvgPool2d(1),nn.Flatten(),nn.Linear(64, 10),).cuda()
© THE