Recently, a team of teachers from Hong Kong has proposed ELF (Embedded Language Flows), a language model modeling in continuous semantic space, which has achieved remarkable results in text generation class tasks. This has made the continuous distributed language model the focus of discussion: can language, like other continuous models, be generated in continuous space?
Recently, under the guidance of Prof. Seung-yang, Liang Jae-yong, of the Nanjing University Model Recognition Laboratory (PRLab), United South Singapore University of Technology and the British Imperial Institute of Technology (UKIT), AURORA-LM (Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling) was developed to further reflect and discover language modeling in continuous space. All of AURORA-LM's experiments were completed at the elevated NPU and further expanded to a parametric scale of about 1 billion, validating the feasibility of continuous diffusion language models on domestic AI computing platforms.
AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling
The paper:
The main page of the project:https://aurora-lm-project.github.io/
Code address:https://github.com/fyv587/AURORA-LM
Where is the difficulty of modeling language in continuous space?
The language is composed of discrete tokens, and even if the process of generating is transferred to continuous space, the end result still needs to be mapped back to the token from a continuous latent sequence. Thus, this sequence must retain enough information for the decoder to accurately map back the complete text; nor can the model be confronted with an overly difficult to learn distribution.
AURORA-LM breaks the issue down into two steps:
The first step is to construct a sequence of continuous vectors of semantic information that can be decoded sufficiently well for text by means of a fully retrained self-encoder;
The second step is to learn the generative distribution of the encoded vector sequence by using a specially designed cluster block factor dispersion model.
At the same time, AURORA-LM is designed for noise input, training processes and small-step sampling, ensuring stable and efficient modeling in greater semantic space.
The model structure
AURORA-LM will break down the continuous language generation into two phases. In the first phase, AURORA-LM first constitutes a continuous latent sequence of text organized in front-end order. As the latent position moves backward, it gradually lengthens the token forwards that it can read; in turn, when decoding a Token, the model can only use the corresponding latent forwards. The continuous latent sequence retains the structure that corresponds to the text running from left to right.
Encoder - decoder to restore the original token for training purposes:
Among them,
It's the first
The probability score of a token outputs a location.
In the course of training, the model randomly discards some of the tokens to ensure decoding is robust.
The second phase modeled the generation distribution of the latent sequence by means of a split-block cause-dissemination model. Aurora-LM divides the latent sequence into several consecutive blocks, using a causal attention mask from left to right between the blocks and the blocks, while modeling within the blocks is done through a two-way attention mechanism. For each noise-induced latent block, the model directly predicts its corresponding net latent; after all the blocks are predicted, the calculation of training errors in the complete latent space:
Among them,
It's a clean latent encoder.
This is a prediction of the spread model.
The model is always aimed at a complete clean latent.
When reasoning, the block is reasoned in order from left to right between the block and the block, and the results of the forecast are repeatedly calculated through the kv cache; different locations within the block can be used with a two-way attention mechanism, making joint noise and synchronous output:
Among them,
It's a latent block that's currently being generated.
It's a frontal lobe that has already been generated.
In addition, the AURORA-LM prediction of current noise-laden latent blocks can use additional information from two different sources. For unconditional generation, the model passes the current latent block back to the next step in the clean latent predicted in the previous step, providing its own trajectory information for subsequent prediction; for conditional generation, the given prompt text is first converted by the encoder to a clean latent forecast, providing external semantic conditions for generating the text, thereby strengthening the model's compliance with the context.
Why keep high capacity latent?
The term "high capacity" refers mainly to the greater dimension of each latent vector. Higher-dimensional vectors retain more rich meaning for text information, making words, syntax, and local order more easily decoded when disturbed by noise.
Figure (a) shows that after increasing the dimensions of the latent vector, it can still retain more decodable text information when disturbed by noise; Figure (b) shows that the quality of generation is also improved after the distribution of different width calibration training noise.
However, higher latent capacities also come at a cost: the scattering model needs to restore a complete clean latent from a noise-bearing state, and the same dimensions are higher, making learning objectives more complex.
Through experiments, the team found that two key strategies had a greater impact on model training efficiency: (a) as shown in Figure 1, the team found a moderately narrower band noise input dimension (bottleneck), which greatly improved the quality of the final model text generated, and (b) showed that improving the likelihood of input to a high noise state during training can further improve the output result.
How to avoid accumulating errors when taking a few steps to make noise?
In addition, the research team further explored how AURORA-LM can suppress the accumulation of errors along the noise trajectory when making little noise. In training, the model is faced with an independent sample with latent noise; in actual production, it must be continuously advanced along its own predictions. If the judgments in the adjacent noise state are inconsistent, errors may accumulate gradually.
For this purpose, Aurora-LM introduces trajectory consistency: during training, the model first obtains the corresponding net latent estimate based on the current noise-induced latent, then moves to the adjacent lower noise state, and binds the adjacent net latent estimate to be consistent.
This update process can be written as:
Among them, the adjacent lower-noise state represents the stop gradient. The post-update state is given to the EMA model to predict the clean latent again. The model estimates the difference between the current state and the two post-update clean latent estimates as a loss of trajectory consistency to bind the two predictions to consistency.
In training, Aurora-LM will optimize flow-matching loss with this consistency loss:
One of them is the weight control of the loss of consistency.
Experiments have shown that trajectory consistency improves results in all measured generating steps, with the most noticeable improvement in fewer generating steps.
More data, verification of results on a larger scale
The team first aligned the ELF-B test setup, using OpenWebText and Xsum as training data under a 130M scale model, to compare the two tasks in the form of free-generation and conditional abstraction, then to extend the split-cause scattering model to 1B parameters, and based on more extensive open-source data training, to validate the effectiveness of the current strategy on a larger model scale.
Free generation: In OpenWebText, AURORA-LM is compared with self-return, discrete spread and other continuous generation models. AURORA-LM's Gen-PPL is down to 23.56, and MAUVE reaches 0.890, achieving the best results on both indicators.
Conditional summary: In the XSum contrast shown in the paper, the ROUGE-1, ROUGE-2 and ROUGE-L of AURORA-LM were 36.6, 13.4 and 28.9, respectively, the highest of the three results in the table.
1B scale verification: The team further expanded the scattering structure to about 1B parameters, based on open-source pre-training data with cumulative training calculations of about 1500 EFLOPs. (b) In the evaluation of nine language benchmark tasks, including common sense reasoning and reading comprehension, the average score of AURORA-LM-L was 32.6; AURORA-LM-L scored higher on all nine tasks in the paper table compared to a larger, publicly available, parameter-scale continuous language model.
What is the meaning of this verse?
AURORA-LM first learns the high-capacity sequential latent of a re-mapped token, and then lets the block-causal spread model model model model its distribution. The inputs, training and sampling designs made around the latent allow continuous distribution generation to be able to decode sequences with discrete tokens, and provide a path to further exploration for longer contexts and larger-scale sequential language models.