Unified Multimodal Models (UMMs) typically use different visual representations to understand and generate respectively: using ViT to extract the semantic characteristics needed to understand, and VAE to provide the sub-variables needed to generate. Thus, the same image is entered into the context with two tokens, and the visual context changes with it; this dual-flow interface is also different from the single visual flow commonly used in visual-language models (VLMs) and is difficult to follow directly along mature VLM pre-training processes.
Recent pixel-space modeling offers another path: work such as JiT shows that image generation can be done without relying on VAE, directly on a pixel. However, it is not easy to extend this paradigm to video: video generation typically relies on 3D VAE, and video understanding is typically handled step-by-step with image encoders, both of which represent different time dimensions, and how to design understanding and generating shared video interfaces remains an open question.
To this end, Invisalign and the University of Waterloo have proposed PixelUMM, a non-coding model that integrates images and videos directly into the pixel space to understand and generate them.
PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation
The main page of the project:https://nv-tlabs.github.io/PixelUMM
The paper:
Open source code:https://github.com/nv-tlabs/PixelUMM
The open source model:https://huggingface.co/nvidia/PixelUMM
The paper was written by Xuanchi Ren, Bryan Chu, Weiming Ren, Hua Ling, Xiahui Huang, Laura Leal-Taixé, Sanja Fidler, Wenhua Chen, Zian Wang, Jay Zhangjie Wu.
Author: NVIDIA, University of Waterloo
PixelUMM cuts images into spatial patches, cuts video into time and space tubelets, and uses this representation to understand and generate, and only through a linear projection accesses the original pixel to a shared multi-modal mainstream network. The mainstream adopts a Mixture-of-Transformers architecture, understanding and generating a set of parameters, but sharing the same attention. The model returns text predictions, while extending JiT-style pure pictorial predictions to video, using flow matching of pictorial space to produce images and videos.
Experiments showed that PixelUMM achieved competitive performance in both image and video understanding and generating tasks. The paper also conducted empirical studies of key design options such as decoder design, space-time patch sizes, and provided a reference for future pixel-space unified multi-module models.
1. Pixel Space video generating effects
The following video is generated by an open-source PixelUMM model: 832×464 resolution, 96 frames (4 seconds, 24 fps). All pixels are output directly from the model, without any VAE decoder.
Scroll to the left and look.PixelUMM's image generation and understanding and video understanding:
2. Model architecture
PixelUMM is a native unified multimodal model that works directly in pixel space, using neither a pre-trained visual encoder nor a VAE-based subspace tokenizer. It uses a Mixture-of-Transformers (MoT) architecture: understanding experts responsible for text prediction and visual understanding, and generating experts responsible for image and video generation.
Image is divided into non-overlapping p×p patches and video is divided into τ×p×p tubelets. That is, the image is 2D patchified and the video is 3D patchified over time, height, and width.
PixelUMM uses p=16;τ=4: Each image patch is 16×16×3 = 768 numeric values, each video tubelet is 4×16×16×3 = 3072 numeric values, and all are mapped to the hidden dimensions of the Transformer through a simple linear layer.
Img_und_linear_proj / img_gen_linear_proj, video_und_linear_proj / video_gen_linear_proj): Clean images and videos are generated by understanding projection input and by generating noise-induced projection input.
At the output end, each mode uses a simple linear head to achieve linear projection for RMSNorm with a layer of reverse direction, respectively producing a 16×16×3 image patch or a 4×16×16×3 video tubelet, and then unpatchify it as a pixel. The output projection is initialized before multi-mode training.
No explicit time-step conditions. PixelUMM removed explicit time-step embedding and time-step-conditioned AdaLN. The de-noise network only receives add-on pixels, with no separate time-step input, and the model deduces the noise intensity from the pixel itself.
Mixture-of-Transformers is primarily pioneered by PixelUMM using Qwen3's decoder-only Transformer to create token-level routing between understanding experts and generating experts. Each block has its own unified layer of routing, QKV/output projection and FFN, and all tokens participate in the same multi-modal self-attention.
Unified multi-module sequence. All four core tasks are serialized as ChatML dialog formats, visual input and generating targets occupy the original pixel token space in the conversation: understanding tasks using clean visual input, generating tasks using noise-intensive targets. Above: image understanding ((I2T) and video understanding ((V2T); below: text graphics ((T2I) and text video ((T2V).
Broad Causal Attention. The text inside is Causal Attention, each image or video block is a two-way attention inside, the back block can see all the blocks in front. When generated, the buzz target can see the prompt and the entire target block, while the front context cannot see the target. Unlike BAGEL, the sequence contains only text tokens and original pixel tokens, with no VAE and ViT characteristics.
The pixel portion is based on the JiT design: the model directly predicts the clean pixel (x-prediction), but the loss is calculated in the velocity space (v-loss). The total loss is the gain of the loss of three pieces of text, images, and video, with different weights used at each training stage.
3. Empirical Findings
Experience with pixel unified model
The paper revolves around eight sets of control experiments (F1 and F8) that summarize the design experience of pixel-space unified modeling.
F1: Image patch size
Under the same sequence length budget, 32×32 patches can process four times as many images per step, but 16×16 patch image generation losses are always lower. Stronger space compression makes image generation harder to learn.
16×16 and 32×32 patch images generate training loss.
Both patch sizes generate results when training 6K, 10K, and 15K steps.
F2: Video patch size
Compared to p32/t4, p32/t2, p16/t4, p32/t1, four tubelet configurations: the stronger the time-space compression, the higher the loss of video generation. The final model uses p16/t4, which is consistent with the compression ratio of common video VAE such as Wan 2.2.
Video generating training loss with four tubelets configured.
Four configurations are generated under the same prompt.
F3: Patch is a fake.
When linear output heads are used, patch grids such as the sky, walls, and smooth areas can appear slightly flatter. The higher the CFG, the more apparent. The paper compares three convoluted output heads: sampled on a Wan-shaped tube + the boundary of the convoluted tube is cleanest, but the training loss is higher and the picture is more blurred; the PixelShuffle achieves the best balance between training loss and pseudo-shadow suppression. It should be noted that the open source model still uses linear output.
Linear output heads left in a smooth area.
Three types of convoluted output heads.
Training losses for each output head.
False shadow intensity of space patch boundary and time tubelet boundary.
The same prompt enlarges the contrast of each of the output heads below.
F4: Pixel space and VAE space
The gradient frequency of the two is close, and the training dynamics are similar; pixel space training occasionally has peak losses. Both losses are measured in different spaces, and the numerical differences do not indicate that pixel space learning is faster.
The number of training losses and gradients in pixel space and VAE space.
F5: Model size
The 8B model achieves the same number of training steps required for loss, about 1/3 of the 1.7B model.
1.7B and MSE and CE losses in Model 8B.
F6: Calculating scale
After the GPU was increased from 8 to 128, the convergence rate for text CE decreased by about 9 times, while the convergence rate for pixel MSE decreased by only about 1.3 times.
8 card and 128 card trained MSE and CE loss.
F7: Multimodal Context Conditions
PixelUMM sends clean image and video conditions directly to an understanding expert, which enables graphical video and video editing without the need to process the two sets of encoders, ViT and VAE, as in BAGEL. At the same time, the understanding of PixelUMM does not systematically decline after these conditions are added to the task of generating.
Conditional images and videos are accessed by understanding experts, and target videos are generated by generating experts.
PixelUMM and BAGEL are contrasting in terms of visual conditions.
Understanding indicators before and after multitasking.
A video of the incident.
Video editing examples.
F8: Video understanding of interfaces
PixelUMM supports two types of video input:
dense_mode is sampled at 4 FPS, with one tubelet for every 4 cylinders;
The sparse_mode is sampled at 1 FPS and is input separately. The resolution budgets for each sparse_mode are the same.
The comparison results showed that the higher dense_mode yielded no consistent improvement: MVBench increased by only 0.07 points, Video-MME and LongVideoBench decreased by 0.22 and 0.68 points, respectively, and LVBench remained flat. That is, the tubelet input of 4 FPS was equivalent to the step-by-step input of 1 FPS.
It's the opposite of video understanding interfaces.
4. Benchmark
The paper reports the results of open-source PixelUMM models (i.e. checkpoints after Joint Stage 2 training) on image understanding, video understanding, image generation and video generation. The overall performance of PixelUMM is comparable to the baseline. It should be noted that the training data for each model is different, and these results cannot indicate which architecture is better or which model converges faster.
Image comprehension. PixelUMM uses the official LMMS-Eval protocol to evaluate. In counting tasks, PixelM achieved the best results in the table on CountBench ((94.30) and PixMo-Count ((73.03)).
In the unified model in the table, PixelUMM scored best on all four benchmarks: MVBench ((70.53);; Video-MME ((57.33);; LongVideoBench ((59.61) and LVBench ((40.41)). PixelUMM uses sparse_mode ratings, with a maximum of 96 samples, Video-MME does not use subtitles; the scores of other models are taken from their respective original papers.
Image generation: PixelUMM scored 0.77 on GenEval, up to 0.83, with BAGEL's long prompt rewriting; on DPG-Bench, the total score was 85.74, the equivalent of other uniform models.
PixelUMM on VBench has a total score of 83.24 ((Quality Score 84.10, Semantic Score 79.80), which is close to other unified models; including Imaging Quality ((67.63);; Temporal Style ((25.84); and Overall Consistency ((27.90)), three of which ranked first in the unified model.
5. Summary
Using single-layer linear projection, the PixelUMM integrates image and video understanding and generation without the use of ViT and VAE, and achieves performance comparable to advanced baseline. Experiments also show that weaker time-space compression typically results in lower generation losses, and that convolutional output heads can suppress patch distortion.
The future can explore more deeply integrated understanding and generated MoE architectures, as well as larger patches and longer tubelets.
Introduction by the author
The first author of this article is Dr. Yang Shou, a third-year student at the University of Waterloo and a tutor for Professor Chen Wen-ho, who completed this work during his NVIDIA internship.
© THE