This could be the moment AMD has been waiting for a long time.
Recently, Wafer AI deployed the Kimi K3 on the AMD MI355X. The result was that 16 NVIDIA B200 models were required to run across two servers to be deployed on one AMD server with eight MI355Xs.
More importantly, it didn't just put the model in.
In a test of 1024 Token input and 400 Token output, MI355X ran a total throughput of 952 Token/s, with a single user generating speed of 118 Token/s.
According to single-node calculations, its throughput is about 3.8 times that of the 16-car B200 program, and its price ratio is higher than that of the B200 and B300.
And most surprisingly, the ROCm was not particularly folded this time.
The model is too big, and it's starting to be more important than computation.
With 2.8 trillion parameters, the Kimi K3 requires more than 1.5 TB of cache in model weight alone, not counting the KV Cache needed in the context of millions of tokens.
An 8-card B200 server, with 192 GB of storage per card, has a total capacity of about 1.5 TB. That is, it is difficult to fully load the model, let alone leave room for KV Cache. Therefore, the B200 must use two servers and 16 GPUs.
The B300 has 288 GB of storage per card, which can be modeled in a single node. The AMD MI355X also has 288 GB of storage, 8 MI355X combined for about 2.3 TB, one server is enough.
This is not just the case with a machine. After the model runs across nodes, each Token generated may need to be synchronized through the network. Even with a RoCE v2 network of about 195 Gb/s, cross-node communication will still delay decoding.
MI355X relies on a larger display to keep the entire model in one node.
From the final results, the peak total throughput of 8 MI355Xs reached 952 Token/s, with a single-track generation speed of 118 Token/s.
In contrast, 16 B200 binary nodes deployed a total throughput of 498 Token/s, in return for a single node of about 249 Token/s.
In other words, the single-node throughput of the MI355X is about 3.8 times the average single-node throughput of the B200 dual-node deployment. In terms of single-user generation, the 118 Token/s of the MI355X is also higher than the 90 Token/s of the B200.
The B300 continues to be the absolute highest-performance solution. The total throughput of 8 B300 nodes reaches 1568 Token/s, with single-track generation speeds of 172 Token/s, which is about 1.65 times the overall throughput of MI355X.
But the price changed the conclusion. Wafer calculated $2.5 per hour for each MI355X card, $4.25 for the B200 and $6 for the B300.
Under this price assumption, MI355X can provide peak throughput of about 48 Token/s per dollar; B200 is about 7 Token/s; B300 is about 33 Token/s.
The B300 is faster, but the MI355X is more cost-effective. For data centers that require large-scale open models, this may be more important than simply competing for a performance championship.
What's even more surprising is that ROCm can basically be used directly.
For a long time, the biggest problem with AMD data center GPUs was often not hardware, but software.
The same model can run directly on CUDA, to ROCm, and may need to change frameworks, add algorithms, or even rewrite the underlying kernel.
However, Kimi K3 is a different story.
AMD provided almost synchronous support for it from the outset. Wafer said the model could be run directly on the MI355X, with follow-up work focused on a few compatibility issues and performance optimization.
One problem arises in the guess decoding link. The Kimi K3 itself does not provide the draft model parameters required by MTP or EAGLE, so Wafer uses an external block scattering draft model.
The program can run directly on CUDA, but in the ROCm environment, the first real request results in a router error.top_k_renorm_probIt's a function.
What this function does is not complicated: it selects the highest value of k from the probability distribution, reduces the other probabilities to zero, and reconstitutes the remaining probabilities.
Wafer ultimately supplemented this logic with a standard PyTorch function, without the need to manually write the GPU kernel, and without the need to redesign the deductive decoding system.
After repairs, deductive decoding improved single-track performance by about 2.2 times, median concurrent single-track performance by about 1.7 times, and peak total throughput by about 18 percent.
More importantly, the system is able to reach peak throughput at higher simultaneities.
The beginning was too slow, only four zeros were added at the end.
Of course, throughput isn't the whole reasoning of the service. For real users, another indicator that directly affects the experience is TTFT, which is the wait time between sending the request and seeing the first Token.
In this case, the initial performance of the MI355X was not good. Faced with a cold start pre-filled task of about 172,000 tokens, the MI355X took about 51 seconds, while the B300 only took about 23 seconds.
In a model that supports millions of tokens in context, pre-filling tasks can be very large. If the context is processed in the long run, users will have to wait tens of seconds or even longer each time, and high decoding speeds can be difficult to compensate for experience problems.
Wafer eventually found that the performance gap came almost entirely from a single focus core. The Kimi K3 would split each GPU into 12 focus heads when configured in parallel with 8 path volumes. AMD AITER's faster MLA pre-filled core supports only multiples of 4, 8 or 16 shapes.
The 12 heads could not match, so the system reverted to the slower general Triton implementation.
The solution is simple: replenish the 12 attention heads from zero to 16, call the existing high-speed cores, and take back the 12 heads that are really needed when the calculation is complete.
After optimization, the AITER MLA kernel's stable pre-filled speed reached about 130,000 Token/s, while the original Triton return path was about 4,000 tonnes to 7,000 Token/s, reducing the cold pre-filled time by about two to three times.
This optimization will not change the final decoding throughput, but will significantly reduce the time the user waits for the first word to appear.
This also suggests that the software gap between AMD and NVIDIA appears to be large, sometimes not underlying capabilities, but that the existing high-speed kernel does not temporarily cover some form of new model.
CUDA's Fortress River is still there, but the gap is already there
One test, of course, does not prove that AMD is fully behind NVIDIA.
The B200 was forced to run across nodes due to insufficient storage; the absolute performance of the B300 remained superior; the toolchain, framework support and developer ecology of the ROCm remained inferior to CUDA.
But open models are rapidly entering the era of trillions of parameters. When a model is larger than a server that can't be installed, display storage capacity is no longer just a number on the parameters table, but directly affects communication costs, deployment complexity and final throughput.
AMD's strategy to configure more HBMs for single cards is becoming a real system advantage.
If AMD can continue to improve the stability of ROCm, expand the shape support for high-speed cores, and provide a more timely first-time adaptation for new models, data centers will have to seriously consider these GPUs.
How do you feel about this?
The link is as follows: