Paper information
Title:ViT^3: Unlocking Test-Time Training in Vision
Authors:Dongchen Han, Yining Li, Tianyu Li, Zixuan Cao, Ziming Wang, Jun Song, Yu Cheng, Bo Zheng, Gao Huang
Institutions:Tsinghua University、Alibaba Group
Readings
This method redefines attention mechanism as an online learning problem that can use key-value pairs to construct a concise internal model when testing. This design method not only makes the model structure more flexible, but also maintains computational complexity at a linear level. However, it is still challenging to design efficient visual sequence modeling schemes: for internal models and training processes, there is a lack of understanding and practical guiding principles for the system. To compensate for this inadequacy, this text provides practical proof of the system's methodology when testing sequence modeling. Through a series of experiments and analysis, we came up with six important principles of image design, which help us to develop and evaluate more effective visual sequence modeling techniques in our design, and ultimately show that we can use T-Vi, T-V, T-V, T-V, T-V, T-V, T-V, T-V, T-V, T-V, T-V, T-V, T-V, T-V, T-V, T-V and T-V. Its performance has always been comparable to, or even better than, advanced linear complexity models (such as the Mamba model and various linear-based attention mechanisms). In addition, it has effectively narrowed the gap between highly optimized visual Transformer models. We hope that this study and the ViT model proposed will yield positive results.
Demonstration of Effects
Softmax attention, linear attention and a diagram of training modules during testing. (a) Softmax attention can be thought of as constructing a two-layer MLP, directly using uncompressed keys K and V, in which the hidden layer width equals the sequence length N and the non-linear function is Softmax. Although effective, this N-width MLP can lead to the computational cost of processing Q ∈ R^{N×d} queries. (b) Linear attention can be constructed by matrix multiplying the linear weight of K^2VK, V ∈ R^{N×d} by matrix multiplying the linear weight of d×d, achieving O N) complexity. However, the linear state of d×d, such as the linear state of a single T ∈ R^{N×d} and the simplified T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T ∈ T T ∈ T ∈ T ∈ T T ∈ T T T T T T T T T T T T T T T T T T T T T T T T T T T T T T T T T T
References
Visual Transformer has become the cornerstone of modern computer vision, driving the most advanced results in image classification, generation, target detection, and division. However, its fundamental limitation lies in Softmax's attention to the secondary complexity of sequence length O (N^2).
In order to break through this second barrier, the research community has explored an alternative to linear complexity O (N). A representative approach is linear attention, which replaces Softmax with a linear nucleus. This allows the computational sequence to be rearranged from QK^T) V to Q (K^T V) to achieve the required O (N) efficiency.
In particular, TTT reimagines the entire attention operation as an online learning process: in each of the above contexts, a compact internal model is trained or constructed from a key-value pairing and then applied to each query. As shown in Figure 1, from this perspective, Softmax attention can be interpreted as constructing a hidden dimension as N and the activation function as Softmax's two-layer MLP, while linear attention is constructing a linear layer of dxd modules through a matrix multiplied by K^2V. Unlike these fixed designs, TTT allows internal models to be arbitrary modules F_W: Rd → R^d.
However, the flexibility of TTT itself also introduces a key bottleneck: a vast and unexplored design space. The lack of a systematic understanding of the internal training process and the internal model architecture in critical choices, and the lack of systematic understanding in existing work, hampers the design of powerful visual TTT models. This gap raises our core research questions: which design principles can be established to build efficient and expressive test-time training modules, and how to promote future improvements?
In order to answer this question, we systematically explored the new design space for training models in visual testing. We focused on two basic aspects: internal training settings (loss function, learning rate, batch size, generation times) and internal model design (architecture and model size). Through a series of experiments, we summarized experience observations into six practical insights: 1) to make ∂2L/∂V∂V ≠ zero loss function L not applicable to TTT; 2) single full batch (batch gradient) internal training is effective in visual tasks; 3) greater internal learning rates that we use 1.0) are effective; 4) to increase internal model capacity for continuous improvement; 5) that under the current TTT settings, deep-level internal model optimization is difficult; and 6) that architecture is particularly suitable for internal model collaboration tasks.
Under the guidance of these findings, we proposed a vision test-time training model, a simple, synchronizable, pure TTT architecture designed specifically for modeling visual sequences. We evaluated ViT3 on a wide range of tasks, including image classification, generation, target detection, and semantic segmentation. ViT3 consistently matched or exceeded advanced O (N) models (such as Mamba and linear attention variables) while effectively bridging the performance gap between highly optimized O (N2) visual transformers. These results confirmed the excellent efficiency and capacity of our proposed ViT3 and demonstrated the potential of test-time training methods in visual tasks.
Major contributions
Our main contributions and highlights are as follows:
• We have conducted systematic, empirical studies of training design when testing in the field of vision, covering internal training mechanisms (loss function, learning rate, batch size, rotation number) and internal model design (architecture and model size).
• We provide six practical insights for building effective and efficient TTT modules and detailed analysis of the TTT design space. Our analysis also reveals several valuable research directions for the future of the TTT model.
• We have built a visual testing training model, which is a simple TTT architecture to implement the above insights. ViT3 has O(N) complexity and competitive results in image classification, image generation, target detection and semantics, providing a strong foundation for future research on visual TTT models.
We presented a controlled study of visual TTT, which refined six practical insights to provide guidance for effective design and potential future work. Based on these findings, we introduced the Training Model for Visual Testing, a linearly complex pure TTT baseline for benchmarking TTT methods on different visual tasks.
The method
Specifically, for internal training, we use a single full batch gradient decrease, with a learning rate of 1.0, optimized point loss. This simple configuration is both efficient and effective for visual TTT. The internal model contains our two useful modules: a simplified gateway linear unit F1 = FC (x) silicon SiLU (x) and a depth-volume F2 = DWConv (x).
The results of the experiment
Summary & Future of Work
Designing efficient and expressive sequential modeling remains a key pursuit of computer vision. In this article, we explore the design space trained during testing, a promising new approach to constructing scalable models of linear complexity. Through a series of experiments, we have refined our observations into six practical insights that provide insights into the design principles and possible future directions of effective visual TTT. Our findings and analysis ultimately form the vision test-time training model, a pure TTT architecture used to build visual sequence models. ViT3 has achieved competitive results in multiple missions, providing a strong foundation for linear complexity methods.
Readers interested in more experimental results and article details can read the original paper.
This article is for academic sharing only. If there is an infringement, please contact the deleted text.