The first author of the paper is Zhang Yi, PhD student at the Hong Kong University of Technology, and the instructor is Professor Li Qing, whose research focuses on image and video production. To date, several papers have been published at top conferences such as ECCV, AAAI, ACM MM.
In animation and artistic production, the color of the lines is a seemingly basic but highly dependent link to the control of detail. Creators often do not simply paint the lines with color, but also need to understand the identity of the characters, observe the local color constraints, refer to the back and forth styles, and maintain uniform clothing, color, texture and background throughout the sequence. Especially in real work, the control signals are often mixed and appear, such as text describing sparse color hints, character reference pictures, recent or long-term historical clues.
However, existing linescript coloring methods mostly revolve around single or minor control signal designs, lacking a unified set of mechanisms to understand control alignment under multi-conditional inputs. More complexly, there may be conflicts between different control signals. For example, reference graphics require characters to maintain character style, whereas local color graffiti requires a certain region to change color. Therefore, a color system that is geared towards a true creation process cannot simply put all inputs together, but needs to judge how much each type of signal should play in the current scenario.
To address this issue, the paper received by ECCV 2026 OmniColor proposed a unified multi-modal color drawing framework. Its central idea was to divide complex control signals into two categories: conditions that require strict spatial alignment, and semantic reference conditions that do not pursue pixel alignment. Based on these two classifications, the model designed targeted coding strategies, training strategies, redundancy elimination mechanisms, and adaptive gate control modules, thus supporting any combination of multi-modal controls.
Figure 1: OmniColor shows unified coloring capabilities. Given linear drawing L, the model can generate accurate color results through different combinations of text T, color cue C, identity reference I and historical string H.
OmniColor: A Unified Framework for Multi-modal Lineart Colorization
Written by Zhang Xuan, Duo Xuan, Liu Yong, Li Qing
Institutions: Hong Kong University of Technology and Sichuan University
Meeting: ECCV 2026
Link to the paper:https://arxiv.org/abs/2603.27531
The code is:https://github.com/zhangxulu1996/OmniColor
The core idea behind OmniColor
OmniColor categorizes common color input signals into two categories. The first is spatial alignment conditions, including input lines, user-provided color cues, and recent historical trends. Such information and target images are strongly correlated in spatial position, and after-color images need to be accurately aligned with input conditions on boundaries, local colors, overall styles, and geometric structures.
The second class is semantic reference conditions, including text descriptions, character identity reference charts, and long-term historical frames. These signals provide more abstract semantic constraints, such as character appearance, overall color, lens atmosphere, or long-term consistency, but they do not necessarily correspond to the current pixel-by-pixel output.
For two types of conditions, OmniColor designs different coding paths. For spatial alignment conditions, OmniColor uses a dual coding strategy: one path uses a VAE coder to retain color boundaries and high-frequency details, and the other path uses a VLM coder to extract local meaning. At this point, the model can both focus on lines, edges, and the location of columns, and understand that these local areas are semantically belonging to hair, clothing, background, or other objects.
For semantic reference conditions, OmniColor uses only VLM encoding, and compressed semantic tokens are already capable of expressing critical properties, and this design can also reduce the number of tokens, to some extent improving reasoning efficiency.
Figure 2: OmniColor's architecture, with auxiliary signals divided into spatial alignment conditions (blue) and semantic reference conditions (green). Spatial alignment conditions are processed using a binary encoder strategy, semantic reference conditions are processed using only VLM encoders, and TRE modules are used to filter redundant tokens. MMDiT's mainstream network integrates these features through AS-Gate modules to resolve input conflicts. The model is optimized through Flow Matching Loss and Dense Feature Alignment Loss.
Three key modules
After identifying the overall coding path, OmniColor further optimises the accuracy of spatial conditions control, reasoning efficiency and the ability to handle conditions conflicts.
Dense Feature Alignment (DFA): Enhanced space constraint. The use of standard loss generation only in VAE subspace is often insufficient to ensure strict local structure alignment. OmniColor introduces DFA loss, uses DINOv3 to extract the dense characteristics of predicted images and real images, and uses MSE loss constraint binary alignment.
Temporal Redundancy Elimination (TRE): Historical Redundancy Elimination. In animation sequences, adjacent redundancies often contain a large amount of repetitive static content, and after all historical redundancies are fully encoded, the model receives a large amount of redundant tokens that affect both efficiency and may dilute really useful change information. The TRE mechanism divides the reference redundancy into multiple patches, calculates the color rectangular similarity of the adjacent redundancy to the patch, filters out highly similar redundant areas, and then extracts the smallest enclosure frames by connectivity analysis to obtain multiple local area images of the changes that occur in the historical redundancy.
Adaptive Spatial-Semantic Gating (AS-Gate): Dynamic processing of multi-conditional conflicts. The difficulty of multimodal control is not only that information is abundant, but that information may contradict each other. OmniColor proposes AS-Gate modules, which adjust the information density dynamics of the spatial branch according to the influence of the semantic branch. Models can reduce the interference of partial semantic references when line drawings, blocks or recent historical frames have given strong spatial constraints; on the contrary, models rely more on semantic references.
Experimental results: improving quality, control and consistency
The paper constructed training data based on 25 high-quality animation sequences and received 120,000 pairs of high-quality samples; the test set was drawn from six previously unseen animations, containing 305 video clips and 900 image-level test samples.
The paper was compared with three types of methods. Under the same input conditions, OmniColor showed higher performance. In addition, the method supports a free combination of control signals, and the same line drafts can be arbitrarily combined with text, color hints, identity references, and historical patches to produce high-quality results.
Table 1: Quantitative comparison with current advanced methods. Note:L = line drawings;T = text description;C = color hints;I = identity reference;H = sequence generation;G = using the first true image of each segment as a reference.
In text-guided color testing, this method achieves text alignment with the linear script structure. In contrast, general multimodal generation models can understand prompts, but are difficult to strictly adhere to the linear script structure; traditional linear script color methods often lack the flexibility to combine multimodals.
Figure 3: Linear contrast in color based on prompts. OmniColor performs better in terms of structural reliability and prompts alignment.
Reference image coloring results demonstrate the model's ability to retain character identity and style information. Even if there is a character rotation between the target image and the reference image, the lens moving closer or further away, and a noticeable change in perspective, the model is still able to stabilize the character's facial features, color and clothing style.
Figure 4: Color quality of the lines drawn based on the reference image. OmniColor maintains a stable identity consistency and style consistency even with multiple movements such as rotation, scaling, and significant changes in perspective.
Multimodal control experiments demonstrated OmniColor's ability to control fine-grained particles. With the addition of an identity reference, individual attributes such as character density and hair color can be more accurately colored; with the addition of a color cue, the model can transmit local color information to the corresponding semantic region.
Figure 5: Multimodal synchronous control. The effect of combining line drawings L and text T with identity reference I and color prompt C.
Serial generation reflects the advantages of this method in continuous generation. When the first string of target segments is used as a spatial reference, the generating effect is further enhanced with the similarity of GT segments.
Figure 6: The sequence generates a visualization display. This sequence uses the generated historical sequence H and the 1st sequence real image G of the target segment.
Finally, the authors tested the real line drawings and obtained excellent coloring results, demonstrating the potential for application to real creative scenarios.
Figure 7: Visual results from real line drawings.
Dissolving experiments
The results of the dilution experiment showed that VLM-only coding significantly reduced reasoning delays; DFA losses increased structural reliability; the TRE mechanism further reduced the token expense of the long-term history gap; and the final addition of AS-Gate achieved the overall best performance, showing that adaptive gate controls can better balance space constraints and semantic references.
Table 3: Merger experiments. #Tokens represents the average token expense.
What is the meaning of this verse?
OmniColor's contribution is to incorporate these input signals into a single set of frameworks, rather than individually training a model or designing a separate flowline for each input. This uniformity enables the model to be adapted to more complex production-level scenarios and provides the basis for the subsequent construction of interactive AI color tools. As future reasoning speeds up, and expands into more abstract sketches or early conceptual design stages, such methods have the opportunity to become high-frequency auxiliary tools in the production of animation, comics, and game assets.
