The Heart of the Machine
BigMac is a new paradigm for running water parallel training in the native multi-modal scenario. It addresses the difficulty of combining computational efficiency and storage occupancy in multi-modal large model training with the problem of relying on secure embedded flow lines: with mature LLM flow lines as the mainstay, under the premise of not disrupting the LLM execution sequence, systematically embedded coders and generators are computed so that the LLM flow line vacuum bubbles remain active while maintaining multi-modal water parallel training.
The next hard battle of the multilingual language model (MLLM) was fought within the training system.
Currently, covering more input models and integrating understanding, reasoning, and generation into the same model system is becoming an important technology direction for leading model manufacturers. At the same time, mainstream training frameworks such as Megatron Core have also begun to support models such as visual, audio, and text, and explore a unified multi-modal input-output training framework.
Modeling capabilities seem to be increasingly unified, while underlying computing is becoming more heterogeneous.
A typical MLLM often consists of three modules: a modular encoder, an LLM master source, and a modular generator, which run on the same set of GPUs but follow different computational rhythms: the encoder converts images or audio video inputs of varying size, quantity, and length into embedding, the LLM master relies on highly optimized streamlines to reason, and the generator maps the output of the LLM back to the target model.
Figure 1: A typical MLLM architecture, which includes a modular encoder, LLM mainframe and modular generator.
At a glance at each module, the industry has accumulated relatively mature parallel schemes. But the problem quickly becomes tricky when these modules are put into the same training stream: a more time-consuming model microbatch, which could cause multiple downstream LLM pipeline stages to stop waiting; and the need to keep the corresponding activation for long periods of time in order to avoid this wait to complete the module calculations in advance, and ultimately hit the apparent storage limit.
Multimodal training is therefore trapped in a definite Pareto frontier: throughput and manifestation often only prioritize one.
The first design is computationally efficient. It separates encoder and generator computation from the LLM pipeline, similar to LongCat-Flash-Omni, which is used to deal with modality-decoupled parallelism in large-scale multi-module training asymmetry. For example, the system may run encoder forward on all microbatches, retain these activations, and then start the LLM pipeline. This can cause the LLM pipeline to be disrupted by the time fluctuations of non-module modules: encoders and generators are slower and won't directly build bubbles within the LLM pipeline.
However, since encoder activation must be maintained until the corresponding gradient returns, activation will appear as the number of microbatches increases. If there are generators in the model, the situation will be worse: LLM activation may also need to be maintained until the end of the generator computation. In production scale models, this is especially expensive because LLM activation is often much larger than encoder activation.
Figure 2: Computational efficient pipeline. This design separates modular computation from the LLM pipeline, reducing bubbles, but retaining a large amount of activation.
The second design is high storage efficiency. It integrates coders and generators into the same pipeline and considers them to be the first stage around the LLM; for example, in the multi-modal training example of Megatron Core, the vision-language model, as a separate pipeline stage, joins the LLM pipeline. This can shorten the activation life cycle and thus reduce storage occupancy. However, all modules are now pipeline-dependent combined. If an encoder or microbatch generator slows down, the Meline pipeline must wait. If the generator's running time changes greatly, the tail of the pipeline produces a bubble.
Figure 3: High-efficiency storage pipeline. This design integrates all modules into one pipeline, reducing storage occupancy, but creating an intermodal bubble.
Small red book dots infra team open source BigMac, which is to solve two difficult problems. Computing efficient systems are fast but costly to store; high-efficiency systems save but easily generate bubbles. For small-scale training tasks, this option may also be acceptable. But for large-scale multi-modal training, especially those that include model generators, this difficulty can be a bottleneck.
BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training
Address of the paper:
This is an open-source address:
The core idea of BigMac
BigMac's starting point is simple: the main source of scheduling should still be the LLM pipeline. Large-scale LLMs already rely on carefully optimized pipeline schedules such as 1F1B or interleaved 1F1B. These schedules are mature, efficient, and deeply tied to production-level LLM training. BigMac does not try to replace them, but instead uses the LLM schedule as a baseline.
BigMac's idea was to first stabilize the most important LLM pipeline, and then insert the coders and generators into the correct location. "Fit" here means that the inputs are ready, and the insertion of these calculations will not disrupt the original LLM execution sequence. Thus, the LLM pipeline can be continued and the activation of the coders and generators can also be released earlier.
Figure 4: BigMac maintains the original LLM pipeline schedule and embeds encoder/generator work in a secure location.
This design has brought BigMac two important qualities.
First, it maintains computational efficiency. Since coders and generators are no longer forced into the LLM pipeline, their time-consuming fluctuations will not be passed along the entire LLM pipeline as they progress gradually as an independent pipeline stage. LLM still operates according to the schedule it should follow in an efficient computing system.
Second, it limits modular activation cache. BigMac will run the encoder backward and the generator backward as soon as possible when the gradient is ready. The encoder does not need to keep activation for all microbatches until the end of the LLM pipeline. The generator does not need to keep a long activation tail. From an algorithmic level, BigMac will reduce the encoder and generator activation cache to O (1), while keeping the LLM activation behavior unchanged.
This is at the heart of the entire pipeline: BigMac is not switching between cache and bubble, nor is it using bubble to replace cache. It changes the schedule structure so that the two objectives no longer have to clash.
Let the multi-modal flood training really land
In real multi-modal model training, the difficulty of pipeline parallelism is often not just "how to sort microbatch", but the framework must also address a range of engineering problems such as system integration, interface definition, performance evaluation.
It provides three key capabilities: transforming the schedule into a visible, verifiable, executable global plan; streamlined wireless interfaces that allow model developers to still write model code in a modular manner; and schedule-aware toolkits that bring profiling, simulation and visualization into the performance diagnostics loop.
1. Put the overall schedule on the front page
The core components of the BigMac runtime are the Scheduler and the Executor, which generates a global operator table covering all pipeline rank, microbatch and module types. This table contains the LLM forward/backward operator, encoder forward/backward operator, generator forward/backward operator, and the communication required at the module boundary. The executor then explains the local operator sequence at each rank and distributes each operator to the back end: the LLM pipeline operator is delivered to the Megatron Core, the model operator is delivered to the encoder or generator runtime, and the data is transferred to the transmission end.
This split gives BigMac two real advantages. First, the scheduling strategy becomes visible and verifiable. We can see directly how the encoder, LLM, and generator can be executed in error across all ranks without having to re-impose global behavior from a long section of the rank-local runtime function. Second, the execution is still backward-friendly. LLM operators can continue to replicate optimized pipeline runtime such as Megatron Core, while model modules can retain their own data-parallel, FSDP, or optimizer implementations. Schedule can also be added to a communication operator for deadlock checks without rewriting the model runtime.
More details are suggested through the official BigMac visualization tool.
2. Let the model code access PP naturally
In multi-modal training, algorithm engineers constantly adjust the connectivity, loss design, and data formats of encoders, LLMs, and generators. The problem is that once traditional PP systems expand into multi-modals, the boundaries of collaboration are easily blurred: a change in the model structure may require rewriting stage splitting and send/recv logic, and system-side optimization may in turn invade the model code.
Figure 5: Pipeline-parallelism-transparent interface design for BigMac. Algorithmic engineers describe the boundaries of the modules, and BigMac processes schedule,handoff and communications at the bottom.
Under this interface, the algorithm engineer only needs to explain what each module produces and consumes; BigMac is responsible for bringing these modules into a global schedule and processing pipeline stage, activation handoff, gradient handoff and cross-device communication at the bottom. In other words, BigMac allows a multi-modal experiment that has already been validated in a single-card or data-parallel environment to be more naturally extended to pipeline-parallel training.
Use schedule-aware toolchain to locate bottles.
The last key capability of BigMac is a set of profiler, simulator, and visualization tools to understand the structure of the schedule. For large-scale multi-modal flowwater parallel training, performance problems rarely depend on iteration time or several rows of log positioning: bubbles may come from a slow microbatch, may come from an encoder/generator running time fluctuation, or may come from an activation handoff, gradient handoff, or cross-rank communication waiting.
Figure 6: Traditional performance screening typically requires downloading and manually associating heavy traces of multiple ranks; the BigMac PP profiler organizes this information into a schedule-aware pipeline diagnostics process.
The BigMac executor records the execution time of each operator and extracts a per-rank timeline/trace, as shown in Figure 7. Through this trace, engineers can directly observe how encoder forward/backward, LLM forward/backward, generator forward/backward and communication operations are executed interchangeably across different ranks to determine whether pipeline bubble is caused by compute imbalance, communication wait, module handoff, or improper microbatch grouping.
Figure 7: Example of pipeline trace generated by per-operator execution time.
The team also provided an interactive PP Profiler trace example, Perfetto UI, rank, timelines and dependencies between different operators.
In addition to the profiler, BigMac also offers a simulator to support faster parallel strategy mutations. The profiler tells why the current training is slow; the simulator tries different PP/VPP configurations, microbatch numbers, module placement and scheduling policies before starting the next expensive training task, anticipating their impact on bubble and throughput.
The results of the experiment
The team evaluated BigMac on two representative multi-modal training loads (MLLM-Understanding and MLLM-Generation).
MLLM-Understanding represents multi-modal understanding training, which uses Qwen3-30B-A3B as the LLM backbone and uses a 1.3B parameter ViT encoder. Compared to the efficient baseline Optimus, the BigMac achieves a 1.08x-1.1x training acceleration; compared to the efficient baseline Megatron-DistTrain, a 1.6x-1.9x training acceleration.
Figure 8: MLLM-Understanding overall performance on training loads.
As the batch size per GPU increases, the BigMac's peak memory stays stable, similar to the high-performance baseline. Optimus, by contrast, retains encoder activation throughout the LLM pipeline; its batch usage grows rapidly and eventually undergoes OOM at a larger batch size.
MLLM-Generation represents training that includes both understanding and generating paths, with the team using the same LLM and encoder, but adding a 20B parameter MMDiT generator. On this load, the difference becomes more apparent. On this load, the difference becomes more pronounced.
Figure 9: The overall performance of the MLLM-Generation training load.
Compared to Megatron-DistTrain, BigMac achieved 1.5x-1.9x training acceleration on MLLM-Generation loads while maintaining stable display usage. This is the most valuable scenario of nested pipeline: the system must handle both encoder and generator dependencies, but cannot withstand large activation retention or severe pipeline bubbles.
Summary
The difficulty of multi-modal large model training is not only that the model is larger or the data more complex, but that different modular modules restrict flow waterline training within an awkward Pareto frontline. The core of BigMac is to break this frontline, which retains the LLM pipeline as a stable mainstay, relying solely on encoders and generators embedded in the LLM execution sequence. Thus, the system can both maintain the computational efficiency of LLM flow and release the activation of the modular modules as early as possible.
What's more, BigMac doesn't just give a schedule, it turns the schedule into a visible, verifiable, executable global program, and it supports PP-transparent interfaces and a schedule-aware toolchain to get the design really into a developmental, programmable, deployable training system.
Multimodal models have opened up new capability boundaries, and training systems need to complement the other half: structured modules can collaborate efficiently across large GPU clusters, and model architecture innovation is no longer subject to high-pressure and flow-throughput utilization. BigMac removed one of the key constraints, which validates one possibility: the conflict between computational efficiency and high-pressure efficiency, rather than switching by one victim.
Author's note
What are you doing?
He holds a Ph.D. in Computer Science from Peking University and has published four papers in top conferences/journals such as TOSEM, TMC, FSE, WWW.
Major areas of research: multi-modal training, distributed training framework
The chapter stands.
Little Red Book dots infra group interns, PhD students at Peking University School of Computer Science, published 9 papers in top conferences/journals in the field of systems such as OSDI, NSDI, sigcomm.
Main areas of research: distributed systems, multimodal models
What are you doing?
He graduated from Carnegie Mellon University with a Master's Degree in Infrastructure Engineering, majoring in Multimodal Modeling, producing end-to-end model training in performance analysis and optimization.
Major research directions: multi-modal/production training, distributed training framework
Presentation by the team
dots infra is an engineering team within the Little Red Book that supports large-scale models of the dots series. The team business covers the core scenarios of large-scale model infra, including text/multimodal comprehension pre-training of large-scale models, pre-training of multimodal generation models, general/application post-training, dot series model reasoning services, data infra, model codesign, etc. There is a hardcore and pure engineering team, with members with top technical backgrounds such as multiple system summits, OI players/ICPC world finals, a robust daily discussion technology atmosphere; and the top computing exhibition spaces in the industry, including world-class Power Co., full-scale self-development technology, ten-million DAU battlefields, Deep-design opportunities.
At the 67th International Mathematical Olympiad (IMO 2026) which just ended last week, the internal version of "dots-note-3.0" scored 7 points in all six subjects, achieving a total of 42/42 gold medals.
A detailed solution can be found here:
Recruitment information
Heating and cooling
Training framework
Responsible for the research and development of LLM/Multimodal Training Framework, optimization of distributed RL training throughput, processing of long sequence / reasoning / CLI Agent tasks
Algebra optimization
Deep into the CUDA/CuteDSL/TileLang basics, to capture performance bottlenecks (such as FlashMLA, Mega Kernel, low-precision algorithms, communication computing overlapping optimization)
Reasoning Engine
Build high concurrency, low latency reasoning architectures, optimize PD/AF separation schedules, unlock architectures and dynamic resource scheduling
Data engineering
In charge of supporting dots (LLM/Multimodal/Agentic) training to align with the ultra-large data engineering system required to cover the entire chain from data production, cleaning, storage, scheduling, distributed reading to quality closure.
Recruitment site