"This is so crazy".
On Friday, the AI community was enthusiastic about a large end-to-end model deployment technology that would allow laptops RTX 4060 to run Qwen 3.6-35B and desktop RTX 5090 to run DeepSeek-V4-Flash 284B.
The median decoding speed of Codex trace production is 33 tokens/s, and the laptop RTX 4060 running Qwen 3.6-35B has already exceeded this number by 39.3 tokens/s.
A joint team from institutions such as UC Berkeley and MIT has opened up a brand new open-source edge reasoning system called FreeToken. Along with Shuo Yang, a Berkeley EECS Ph.D. student, and Xiaoze Fan, both undergraduate students at UT Austin, the authors of FreeToken include top scholars such as Ion Stoica, Matei Zaharia, Kurt Keutzer, and Hanasonic.
The system is designed with full-featured collaboration designed specifically for local deployment of large MoE (mixed expert) models on consumer-grade hardware, breaking the hardware threshold for large-scale front-end models that must rely on data center clusters.
FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution
Link to the paper:https://arxiv.org/abs/2608.16157
GitHub：
The official website:
FreeToken is now available as a Windows/Linux desktop app (with GUI), and a line of CLI uv pip install "freetoken [accel]" can be installed.
What is consumer-grade hardware?
Traditional CPU-GPU offloading schemes often result in extremely slow reasoning (a few tokens per second or even a carton) because of the PCIe bandwidth bottleneck, while FreeToken delivers local reasoning at "interaction-level practical speeds".
In addition to the speed of Token per second, traditional Offloading programs cause severe I/O blockages when processing the long Prompt (such as 4-16k below) due to the layer-by-layer extraction of Expert from the system memory. FreeToken reduces the delay of the first Token (TTFT) by 42-58%.
For today's common Agent tools, such as Claude Code, OpenClaw, the cyclical interaction, and the gradual addition of contextual scenarios, TTFT for subsequent multi-round interactions decreased by 65-80% due to the lightweight border checkpoint and state replication introduced by FreeToken in the multi-tool call and mind-chain (CoT) iteration.
The paper was tested on different PCIe specifications (PCIe 3.0 x8, PCIe 4.0 x16, PCIe 5.0 x16) and different CPUs (such as Intel i7, AMD Ryzen 9, Threadripper) where algorithms can automatically improve CPU end-to-end distribution computing ratios when other high I/O tasks take over the mainline on the back of the system; and GPU load ratios when the mainline is empty.
In other words, no matter what the broadband bandwidth limiting environment the system is in, the system can be stablely confined to the theoretical maximum throughput limit under the hardware topology.
The paper's authors also performed a stress test to simulate users opening 3D rendering in the background; the game resulted in scenes of 4-8GB of available display storage plummeting. Traditional programs would directly trigger CUDA Out of Memory (OOM) collapse to exit, FreeToken would be able to reduce the size of the LRU Cache in which the CPU resides with zero downtime, transferring more unintended expert computation to the CPU, ensuring smooth degradation of the reasoning service without interruption.
It's easy to say that the running model doesn't have to be a dead-end model.
For rare activation architectures such as MoE, as long as the "dynamic bandwidth perception + dual buffer data flow + state duplication" in software-hardware synchronization is fully exploited by the more common DDR5 memory + PCIe high-bandwidth channels + consumer-grade single cards on PCs, it is possible to get an experience for everyday personal use at a very high price.
Of course, this means that there is not enough memory. For example, the DeepSeek V4 Flash expert pool of about 140GB requires 32GB of RAM (RTX 5090) plus 192GB of memory to run relatively stable.
Why can FreeToken do this?
Today, the big-model open-weight solution is who gets the model without solving "who can run the model". Kimi-K3, GLM-5.2, DeepSeek-V4-Flash's capabilities are close to the closed-source first ladder, but running them still requires a data center by default. As a result, the capability gap is narrowing rapidly and the availability gap is large.
If you look at the Steam platform, about 72% of your computers have N-cards, the most common of which is the RTX 4060 for laptops. If a large model could use them, it would be very different. Then you need a service system that can coordinate GPUs, CPUs, host memory, and each other as a single platform.
MoE happened to give us this opportunity, for example, the hottest DeepSeek-V4-Flash yet, which has 284 B total parameters, 43 layers, 256 routing experts in each layer, six of which are selected, and a single token only activates 13 B. At deployment accuracy, the active parameters fit into 32 GB of the RTX 5090.
The current bottleneck is that Prefill will destroy the scarcity of MoE: a single token only runs k of experts, but a long prompt combines all the token routes to cover almost all of the experts at each layer.
In the deployment of DeepSeek-V4-Flash, the FP4 deployment takes about 140GB of expert weight, and the RTX 5090 requires an additional two seconds, with an on-demand engine that exposes the entire part to GPU vacuum.
Adding to the snow is that Agent changes the context for each round of tool calls. Qwen 3.6-35B uses a gated DeltaNet, Kimi-K3 uses a loop layer like KDA, which compresses the entire front end into an evolutionary state that cannot be replicated locally, only by the checkpoint. A checkpoint has a memory equivalent to hundreds of KV tokens, so the engine can store only a small amount.
FreeToken's solution is to redefine the synchronization scheduling logic of end-to-end isomorphic hardware:
At the Prompt processing stage, FreeToken achieved a complete overlap of computing and data transfer: when the GPU was computing layer l, the Expert weight of layer l+1 had already been pre-loaded through the backend of PCIe, essentially eliminating the I/O waiting bubble.
Bandwidth Adaptive Execution. Real-time detection of the PCIe real-time bandwidth of the machine with CPU instantaneous calculation, dynamically calculating the optimal flow rate q*. Part of the Expert lives in GPU LRU cache, part of the unintended lives in real-time bandwidth intelligent distribution, part of the transmission to GPU, part of the real-time computing in parallel with the CPU, pulling the hardware through to the limit.
Modern Coding Agent or Tool Calling Models frequently minimize/modify context history (such as adding thought steps or tool return results). FreeToken sets light checkpoints at the special Token boundary and only restores and adds Prefill from the nearest focal point when encountering context editing to avoid re-computing the entire context.
In cases where other applications (such as games, browsers, development tools) are often available for personal computers, FreeToken supports the size of the Expert Cache in the dynamic heat adjustment cache if the Serving service is not restarted, ensuring that the service is not interrupted.
The mechanisms for targeting the Agent are very interesting, so it's possible to say that targeting today's hot Agent missions is being optimized.
What is the meaning of this verse?
FreeToken demonstrates that the key to making big models popular is not only the weight of open-source models, but also the engineering design and algorithmic restructuring of end-to-end modeling software systems.
By combining the CPU, system memory, PCIe and GPUs of personal computers as a unified, resilient computing platform, developers and individual users will be able to run the top MoE Agent entirely on their local laptops or gaming PCs, greatly reducing the cost of deploying highly intelligent models locally.
Well, now it's the legendary Qwen 3.8-A3B.
