Think about folding into a sequence: WeLM 617B MoE's implicit scaling path

  It's the first time in the history of the world.

The big model is getting stronger, and in the past, it was two-way.

After the Big Bang Scaling Law, the parameters jumped from 10 billion to 100 billion and computational spending skyrocketed.

I'm looking for a long-term, burning thinking model to replace it with a longer chain of thought and more time to reason.

The question is, is there a third way besides the scaling parameters and the chain of thought?

WeChat's WeLM team gave a whole new set of answers

Fold the additional arithmetic into the sequence, allowing the model to complete implicit thinking between tokens. The output is constant and each token thinks deeper.

This method is called Hidden Decoding.

The key step for WeLM is to push it to the forefront.

From March's 80B parameter to the current 617B MoE, not only did incremental training use only 5.3% of the full training, but nine assessments all exceeded the baseline of return.

On July 14, WeChat's WeLM team released the second blog in the Hidden Decoding series, showing for the first time the progress of a new model: a model with a total parameter 617B to evaluate performance at open-source head-level.

Address of the blog:

This is not the first in the series.

On March 2, the WeLM team published the first blog in the series, Hidden Decoding, a relatively simple early implementation that only ran on an 80B scale.

Four months ago, the new blog not only solidified the approach, but also demonstrated for the first time the effect of pushing the program to 600B+ models.

At the heart of this blog is how Hidden Decoding has been pushed to the front-end scale of 100B+ MoE.

Spreading each token across a sequence dimension into multiple parallel "flows" allows the same set of Transformer weights to perform multi-step implicit reasoning in one forward propagation, while only calculating losses on the last stream.

In other words, the model folds into an unseen thought inside its sequence before writing the next token.

If the two main lines of a large model over the past year are "to enlarge the parameters" and "to extend the chain of thought", Hidden Decoding shows a third path: under the premise of static parameters, extra algorithms are stored in the sequence itself, allowing reasoning to go a few more steps within the model.

Over the past year, the efforts of the big model community can be roughly divided into two lines.

One is to enlarge the model itself, bringing the total parameters to trillions and activating parameters to more than 30B; the other is to build computational power in the post-training phase, using reinforcement learning and a longer chain of thought to "force" reasoning ability.

With the rise of the o1, DeepSeek-R1 and the "Thinking Model", the latter route has become almost the default narrative.

But the two lines share the same implicit premise, and both require "more" -- more parameters, more high-quality reasoning data.

Hidden Decoding's suggestion circumvents this premise. It doesn't do articles on the data side, but goes back to the pre-training and continuing pre-training stages and asks a more simple question.

Could the same set of weights, the same sets of data, be able to force more capabilities through smarter computational organization?

This is not a path that no one has walked.

Cyclic models, as well as various attempts to "extend in sequential dimensions", are all part of the same idea.

The card point is also clear. Circular models are not parallel in sequence, the sequence expansion is easy to allow attention to be squared by the length of the sequence.

So while there's a lot of discussion about it, there's very little that's really going on at the forefront. WeLM's blog post is about re-opening the road that's been bypassed at 100B+ MoE.

WeLM is a generic big language model series developed by the Weibo AI team that carries the core technology strategy of Weibo's Big Model era and is the most critical technology underpinning for Weibo's participation in the front-end AI competition.

Since its first public appearance in 2022, WeLM has continuously generated four generations.

While the team has always maintained a cautious attitude towards external voices, it has clearly outlined a path to technological evolution from recent focused blog posts.

From training systems built in the V3 era, to efficiency and quality innovations in the V4 era, to the Scaling breakthroughs in the 600B model and the reasoning paradigm innovations in Hidden Decoding.

This marks the formation of a comprehensive technical version of WeLM that covers pre-training, post-training and reasoning optimization.

More importantly, WeLM has not stopped at the experimental model of paper indicators and laboratory assessments.

Microsoft AI assistant "Small Micro", which is currently in the gray testing phase, has its core modeling capabilities provided by WeLM. WeLM has successfully entered the final testing ground consisting of real users, real traffic and real complex tasks, based on WeLM's national product ecosystem with a super-large user base.

From the publication of this technology blog in the Hidden Decoding series, we can see how the WeLM team is thinking about combining cutting-edge model technology with scale-up application practices.

Hidden Decoding can be summarized in three steps:

1. Embedded in a given input sequence and n statement table, the expression "flow" in the k-bar of the second token is placed at the physical location ((i−1) n+k, constituting an extension sequence of length nL;

2. The extended sequence is encoded in the same Transformer under standard causal attention and continuous location, without adding any main root parameters;

3. The next token's predicted loss is only calculated on the last stream of each token during training, and the previous n-1 stream does not receive any direct supervision.

In other words, the previous stream was to give the final "drafting" class a gradual refinement of the characterization, retaining a broader set of candidates, etc. The last class would be retrofitted to the final prediction.

Hidden Decoding vs Loop Latent Computation

Previously, the team had verified the loss reduction and evaluation improvements of this approach on 6B, 8B, 80B. This time, the goal was to push it to the true cutting edge scale.

Draw the sequence from L to nL, not paying attention to the approximate linear growth, but concentrating attention to the square expansion of the sequence length.

For example, n = 4 is 16 times the size of a large model of a long sequence, which cannot be taught directly.

The team's solution is Stream-Factorized Attention: allow most layers to focus only within the same stream, and allow only a few layers to mix across the stream.

• In-flow attention: focusing only on earlier positions in the same flow;
• Cross-flow attention layer: Additional cross-flow attention, following the original focus form, sliding window or full attention along the base.

The key is that the team doesn't add a new attention layer, but instead enables cross-flow functioning on the existing local attention layer and a small portion of the full attention layer in the baseline.

This puts the newly increased attention costs from n squared to nearly n times linear. This is also the core of Hidden Decoding that can be found on the 100B+ rare model.

In addition, the team used an already-designed KV-mirror from the WeLM mainframe to further accelerate, i.e. the second-third layer replicates the first-third layer's cache state to construct key value caches.

Since the key values of the mirror layers depend only on the state of the initial layers, and only the last stream is monitored, the middle stream only works through the key values they contribute, the team only allows the last stream to pass through the mirror layer, and the first segment still processes all the n streams.

In the 80B, 32k training setting, the one-step time was reduced from 15 seconds to 12 seconds, an improvement of about two percent, superimposed on the near-linear costs that flow factorization attention has provided.

The team compared two shapes for each base, all the same except for Hidden Decoding. It was then adapted using the same set of short instructions to fine tune the formula without reinforcement learning.

The results show that Hidden Decoding has achieved comprehensive improvements on both scales.

Of these, 80B improved by an average of about 0.99 percentage points, up by about 1.03 percentage points on 617B, with major improvements in more difficult reasoning and knowledge tasks such as SuperGPQA, MMLU-Pro.

What is the cost of training with such a set of methods?

When n = 4, the effective sequence spends 5.1 times the single batch time on 80B; 4.4 times the linear reference time on 617B, close to n = 4, much less than the 16 times expansion of dense attention.

More importantly, Hidden Decoding is not training from scratch, but add-on training at the CPT that has been in place since the return base checkpoint.

Additional calculations of Hidden Decoding are parallel to those of the circular model.

The cyclic model gradually repeats the main sequence, with each step waiting for the next step, and the additional costs cannot be hidden; Hidden Decoding deploys the computation to n streams, which are processed in parallel and forward.

The test of the throughput matrix shows:

• In large batches, decoding is computationally limited, with additional computation reducing throughput (this is the case for all methods of increasing computation);
• In small batches, decoding is limited by the display storage bandwidth, calculating a large amount of latency, and parallel streams use these vacuum computing capabilities, with relatively little loss of baseline.

This means that in the real online service scenario, the reasoning cost of Hidden Decoding is not a simple linear growth, but is strongly related to the batch size and sequence length distribution.

Hidden Decoding relies on an anti-intuitive option: train only the last stream and zero supervision of the front stream.

The main results suggest that this improves accuracy, but does the gain come from "a longer sequence", "more predictive targets", or "the middle stream is actually doing some refinement"?

In the focus blend, the team compared three configurations: 27 layers of cross-flow, 4 layers, 1 layer.

All variants are significantly larger than the return baseline, and full-flow is best, but only one-layer or four-layer full-attention variants recover most of the benefits. A few layers of cross-flow is enough to keep fluid-factoring attention affordable while maintaining accuracy.

The dilution of key value cache retention indicates that independent retaining status is better for different streams than shared status, with both layouts averaging a slight decrease, indicating that independent intermediate status is helpful.

The most interesting was a probe experiment with flow and language model heads. On the 8B base plus Hidden Decoding ((n=8)), the team observed two points:

Second, the flows of the same token are significantly differentiated in the intermediate layer of the Transformer, with the adjacent output layer partially anchored, and the end stream allocates considerable attention to the other flows, forming the reading path from the intermediate stream to the predicted stream.

Second, when the shared output heads are applied to each stream, the optimal prediction of the midstream is often different from the final stream, with a maximum difference of about 63%, and its uncertainty is generally higher than that of the final stream.

Combined, fusion and probes point to the same explanation: a wider set of candidates is retained and refined before the final flow converges to the prediction.

The team believes that this is what it means to "use the forward flow as a subspace computing state" and that Hidden Decoding is key to distinguishing it from simple "enhanced sequences" or "multi-target predictions".

The blog finally revealed a bit of follow-up progress.

After training with higher-quality, larger-scale data, the 80B and 617B have demonstrated significantly higher benchmark performance and a more stable user experience.

This set of data shows that WeLM's self-return baseline is already very competitive, and Hidden Decoding is a further advantage over this baseline.

The team says they will continue to verify the benefits of Hidden Decoding on this stronger baseline and fully integrate it with reinforcement learning.

Looking back at the recent technology blogs released by the WeLM team, you can see a gradually clear, back-and-forth-response technology route.

• Using moderate resources to build efficient, sparse MoE models, Liu explored how to gain competitive ability with more sparse, more efficient models, while squeezing resource consumption at the same level of intelligence.
• The Hidden Decoding series started with innovative reasoning paradigms and continued to push up the limits of intelligence under the same model parameters.

The former makes smart services more user-friendly, while the latter makes limited resources more difficult to solve.

Looking back in two directions, this series of blogs is not an isolated technological attempt, but a systematic exploration of resource efficiency that echoes the theme of "extreme resource efficiency" on the team's blog.

Extreme resource efficiency is not a constraint on smart growth for a WeChat team that has to deal with massive real users and complex tasks, but perhaps another pathway to higher intelligence.

This is my first blog post.

The first podcast:

The article is in PDF format:

It's a bit of a mystery.