Two days ago, a post by Max For AI@MaxForAI, a netizen on X, sparked a viral outcry: "It's so crazy, you can even stream live around a 535B big model being trained".
Professor Percy Liang@percyliang, founder of Simile AI at Stanford University, announced the launch of the Marin 535B-A23B model from Marin's open lab, and the entire training process will be open to the public.
Marin has 535 billion total parameters and 23 billion activation parameters, for which Percy Liang and the team prepared training data for a total of 18.75 trillion tokens and deployed 11 sets of GB200 NVL72 systems, for a total of 792 GB200 GPUs.
The model is expected to train continuously for about 3 months, with a total training calculation of about 2.7e24 FLOPs.
In other words, in the coming months, everyone will be able to see how a big front-end model will grow from scratch.
It is also worth noting that, prior to the official launch of the training mission, the team had already trained a set of Scaling Ladders (scale expansion ladders) consisting of four stages, from 1.6B-A61M (48B tokens) to 27.7B-A1.2B (926B tokens).
"This is done for two purposes: first, to detect and moderate potential problems in advance with these small-scale experiments; and second, to predict our "master model training" ("hero run") based on the training performance of different scale models".
Of course, Percy Liang also said, "This is the biggest training we've done so far, so we did have some unexpected things going on in the process of waiting".
Currently, the main model has been officially launched, and everyone can directly view the training curve in real time. Follow-up, training data, experiment logs and engineering issues will continue to be public.
For example, the data level, including training data mix ratios, how data is processed, and how data in different fields enters the model; the engineering level, including training configurations, code, and experimental design; the training process, including real-time loss, changes, model status, and predictions of results at different stages.
At the same time, Percy Liang also unveiled specific viewing channels.
See how the data is composed:
Watch the training process in real time on Weights & Biases:
See all the details on GitHub:
It's not hard to see why Percy Liang's operation has captured the public's attention after learning more about it. In the past, for models like the GPT-4, Claude, Gemini, it was almost impossible to know exactly how to train the back, like a "black box".
You can only see the final model capabilities, benchmark scores, a small number of paper descriptions, etc. But we don't know what the data comparisons, training failed, which hyperparameters work, how the losses change, whether the scaling laws predict accurately, etc.
Marin is trying to change that. Maybe model training can be like papers, open source software: it can be observed, it can be discussed, it can be collaborated.
Since the announcement of the news, the enthusiasm of netizens has continued to rise.
Some netizens believe that this is the true open-source spirit, "even if the training process has problems, deviations from expectations are not hidden".
Some netizens say that even though the training model itself is huge, what's really shocking is that people can now watch a multibillion-dollar model training process grow step-by-step through the WandB platform in real time.
"It's like inside a top-notch AI lab where the shutter is suddenly open and everyone can see what's going on inside".
Over the next three months, perhaps what is most anticipated is not what the final model is capable of, but how many "explosions", crashes and accidents" the model will experience during training.
In addition, some netizens believe that the move not only provides a perspective on the entire model training process, but also provides a platform for learning and communication.
Networker James Thewlis@jdthewlis has been following the training process in real time, during which he doesn't understand "why is there a peak near the 500th step, norms (parametric parameters)?"
After realizing it himself, he asked himself in the comment section: "This peak was caused entirely by the router_bias routing bias. It is not a parameter trained by gradients, but is used in the MoE hybrid expert model as part of the token balancing heuristic, so it has a dynamic change law that differs from ordinary model parameters".
There are many other situations like this.
In addition, it is worth noting that as a Stanford professor, Percy Liang, in addition to the practice of the real knife, there is a theoretical course: CS336: Language Models From Scratch (Building Language Models from Scratch) which aims to give students a complete understanding of how a large language model is constructed.
It looks like Percy Liang really wants to teach everyone how to make a big model.
Link to the course:
So what do you think of this live show that really moves big model training to the stage?
The link is as follows: