It's the first time in the history of the world.
Recently, Ingrid published an official guide to big model reasoning, and the results were overturned, hardly making it look like a collection of papers by a Chinese team. This is how it is. On September 2, Ingrid's technology blog posted a long article using speculative decoding to co-design AI models.
Its distribution is that it is extremely rare for chip masters to write a specification in an official gesture of "how models should set the hardware physical limits to design". When they look around to find the best way to dance at the chip limit, they include the core program in the guide, almost entirely dominated by the Chinese team. This has already gone beyond the usual algorithmic summary category.Almost pure bullshit: what's up with speculative decodingIn order to understand this table, you need to understand what "speculative decoding" is. The big model spits, one that goes to the outside. For every spit, the parameters of hundreds of GB have to run a full circle in the cache.
Find a small, lightweight model to make a pre-determined prediction, first guess seven words, and then verify the seven words once in a while.
It takes almost seven words to verify and one word to write yourself, after all, the weight has to be moved back and forth.
Guess what's right is taken directly, and guess what's wrong is reversed from the breakpoint. Because the big model only accepts the words that it would have written itself, it doesn't have to change any of the endpoints. This is called "lossless acceleration".
In this game, all the arithmetic pressures revolve around a core mathematical expectation formula: Speedup = E[L] × TtargetTdraft + TThe verify molecule is the gain E[L], which represents the expected length of acceptance (the large model averages a few words per spin). The divider is the extra cost you pay, the small model makes the prediction time (Tdraft) plus the time for the final verification of the large model (Tverify).On the shoulders of one's peers, four generations of evolution unfold.Following Ingrid's chart from top to bottom, you'll see a clear slaughter line. The first line is the oldest "external draft model", which has to train a small model separately as an assistant. Ingrid directly unveils the price bill: from the beginning training to consume 1T to 10T tokens, the cost is very high. However, it remains on the table because Ingrid has arranged a specific homage to the $20 billion Groq chip L. The last line is a non-trained n-gram checker table, which relies directly on the hard copy from the text above, and is only suitable for the dead scene representing the large segment of the paste.
EAGLE-3 -- a team of people from Yuhui Li, University of Waterloo, Hongyang Zhang, and others. From ICML, EMNLP to NeurIPS, three generations went on to physically capture the "word-for-word guess" of this series of old routes. It was once the absolute master of this field, but in Great Britain's new table, it was squeezed into a "reference point", and the reason was very short: acceptance of the length had already been surpassed by the waves. It completely abandoned EAGLE and MTP's "one-word-by-word guess" sequencing method, and instead drew on a spreadsheet model, which once computed forward, directly generated seven Token prediction sequences in a one-time manner, reducing the fractional (time-consuming) to extreme compression. By the way, instructor Zhijian Liu was an assistant professor at UCSD, but he was also a research scientist at MIT.Hardware constants, reverse-locked model architectureThis four-generation technology can make computation dry to this point, relying not only on algorithmic ingenuity. Many people think that the more a draft can guess at once, the more difficult it is, but this completely ignores the physical rules of the chip. When the attention mechanism occupies the bulk of the decoding time, the total Token required for the large model verification phase is 1+D: 1 real input + D draft prediction. In order to pin this portion of the data to the GPU, the hardware will group the underlying query head and the KV head, and the size of each group of attention kernel blocks must be strictly governed by the boundaries of the GPU physical matrix (Tile Size, the current architecture is usually 128).D) ≤ 128 slightly further down, we get the final constant rule in the Invidia Guide:D = 128G- 1 of which, G is the ratio of the attention grouping number (Query head and KV head). This means that your model at the beginning of the design determines directly how to group attention and how the draft should guess a few words to fill the GPU's parts tightly. If G=8, the draft just guess 15; if G=32, guess only 3. The hardware crosses the boundary, even if the last Tile you only use half a block, calculate the same by the entire block. Before, the decoder was thrown out by the model rehearsal boiler, the engineering team then thought about how to hang the accelerator pack. But now, the laws of physics at the bottom tell you: a number of a hardware matrix is directly reversed to the design parameters of the model architecture.
The end.The focus of the competition for the efficiency of the model has completely changed. The era of simply stacking huge parameters is turning around, and today the real downside depends on who better understands the physical limits of the chip. In this new game played by the chip giant, the Chinese team has quietly become the default answer to failure.
RoboInd editorial channelResearch, robotics companies and market intelligence for cross-border sourcing.Explore the channel ↗
Share this
End of editorialExplore more from Global Robotics News ↗