Today's big models are smarter, and the output content is one word at a time.
Recently, researchers at Stanford University and Together AI have proposed a third text-generation paradigm that differs from both self-reporting and differs from widespread text-generation. They moved the best paper VAR of NeurIPS 2024 into text in the formulation of "next-scale prediction" on the image, allowing the model to simultaneously write a "concept outline" of a few sentences, and then fill in the details in layers until the full text was created.
So the new model is called Next-Concept Modeling (NCM, next-concept modeling), and the mainstream Next-Token Modeling (Next-Token modeling) has a positive message.
In several experiments, NCM achieved a production rate of 10 to 31 times that of a relatively widespread language model with similar or better quality.
Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation
Link to the paper:https://arxiv.org/abs/2610.14162
GitHub：
We know that the current mainstream big language model is the self-return model (including but not limited to GPT, Claude, DeepSeek), which is typically based on content that predicts the next token, with subsequent output dependent on the previous outcome. In recent years, the increasingly popular distributed language model has been able to process multiple locations simultaneously, generating text gradually by repeatedly making noise.
The idea behind NCM is to change with "what each step produces". It draws on the next-scale prediction in image generation, which allows the image to be rendered in a low-resolution version and then gradually add details. Since text can be scattered, can the text also have its own "shortcut"?
The image is from the NeuralPS 2024 Best Paper series Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction series.
The difficulty is that this is a two-dimensional thing: the subject can usually be seen after the image is enlarged, and the removal of a few words can completely change the meaning. Therefore, we need to let the model itself learn a representation that is both generalized and suitable for further refinement.
The first is training a multi-dimensional residual VQ-VAE. We can understand it as a set of text coding and restoration systems: converting the original to multi-layered decentralized coding, with less front coding, more back coding, and each layer continuing to add information that has not been previously expressed. In order to prevent the rough layer from "stealing", training requires the system to recover the back layer as much as possible using only the remaining parts.
Subsequently, the system was frozen and the Transformer trained to predict the next layer based on the previously accumulated representations of each layer. When generated, the model progressed layer by layer, coding the same layer at the same time, and finally the decoder restored the text.
What is the effect of the model? The paper shows examples of translations from German to English, where NCM first decoded the middle layer, and the initial output was a noisy repetition and incomplete expression, and as the levels progressed, the content became clearer and the final tenth step formed a smooth translation.
Another feature of NCM is that it is fast, on the one hand, the number of generating rounds that need to be executed in succession is reduced, on the other hand, early only dealt with shorter coding sequences, and the average cost per round is also reduced.
The authors tested on the single block H100 SXM 80GB, with LM1B unconditionally generating about 21 times faster than the strongly distributed baseline, generating better confusion; OpenWebText's quality is competitive, at about 10 times the speed; in the XSum abstract task, at about 31 times the speed, at the equivalent of ROUGE-L. The two datasets use 4.9 times and 4 times fewer tokens, respectively, at training cost, while the DUO and MDLM objects run 1 million steps on OpenWebText and process more than 50 billion tokens.
In reasoning tasks, for example, NCM outperformed the regression and scattering baseline in fewer sampling steps. In a difficulty problem with only 30 clues, NCM's complete resolution rate was 86.1%, higher than FMLM+'s 81.3%; the number of generator calls was 12 and 16 times, respectively.
Gradually refining the entire chessboard might be better suited to coordinating the overall constraints.
Looking ahead, the speed can also be used in a different way: to generate several candidates at the same time and then select from them. The team tried decoding the minimum Bayes risk, using the similarities between the candidates to select, and further improved the translation and abstract indicators.
Of course, away from the general large model, NCM still has many questions to answer. Generators used in the paper subject experiment are about 100 million parameters, and the scale is expanded to 725 million parameters.
If text generation can also be represented as a layered structure of "conceptual detail", models can first think about penciling, which is also the way humans write.
This means that future text-generated roadmaps may be rewritten.
References are given below:
© THE