Today's top AI is indeed powerful, and new developments are continuing to emerge, especially in mathematics, and it has even conquered a millennial problem.
However, top-level AI isn't cheap either: the Astra API standard costs $10 per million tokens to input and $50 to output; the Fast Mode charges twice as much as the applicable rate. If a single request to input more than 272,000 tokens, the entire request will be charged in context, with the price of input doubling to $20/million tokens, and the price of output rising to $75/million tokens!
In contrast, a 4B small model can cost as little as $0.08, which is hundreds of thousands of times higher than top-level AI.
As the ceiling of capacity rises, it is not enough to call for a single price. So the question arises: what should we do in the face of increasingly inflated bills?
Pyromind's recent open-source PyroDash gives a somewhat counterintuitive answer: let the small model decide for itself when to call the big model.
PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference
Technical report:
And they have demonstrated the effectiveness of this strategy in practice: on five mathematical reasoning benchmarks, small models are called large models on average for every 100 questions when they are more cost-effective, and the estimated total cost decreases from $49.36 for pure models to $1.78, less than one-third of the original, with little reduction in accuracy; and if accuracy is more important, the combination of small models + large models averages an accuracy of 64.04%, which is 6.36 percentage points higher than 57.68% for pure models, and costs two percent lower.
Total cost and average accuracy of five mathematical reasoning tests
According to Kevin Ding, founder and CEO of Pyromind, "PyroDash originally came from our own needs, because Vibe Coding was too expensive". The team began studying small-scale model collaboration in an automated test loop to continually pursue code quality.
And Pyromind isn't just about saving money, it's about judging a bigger route. In a recent podcast, the company's founder and CEO, Kevin Ding, summarized a selection of discussions about the "second half of AI" from the second half of 2025:
One path is to continue down the pre-training path, and the ASI finally comes in an extremely large, centralized model form that solves the vast majority of the scenarios and works well enough;
One way is to make up the ultimate ASI, a "Agent beehive" with a large number of different levels of AI intelligence.
He chose the latter path, arguing that "the problems and scenarios in our world that need to be solved by AI, regardless of the current point of view, are already endless". New scenarios are continuing to emerge and the scenarios themselves are not static.
From Crossing the Crossroads
Peter Belcak et al. of the Institute for Advanced Study in Ingrid published a position paper last year suggesting that small language models are the future of intelligent AI, arguing that most calls in agents are in fact repetitive and formatted narrow tasks, that small models are competitive, fitter, or more economical, while mixing heterogeneous systems of multiple models is a natural choice where there is a real need for general conversational ability. Earlier, Matei Zaharia et al. of Berkeley BAIR blog posted that "the shift from model to integrated AI systems" was the most forward-looking result, increasingly coming from multi-component systems rather than single-component models. As of August this year, all Gartner analysts Sommer also said that in the foreseeable future, a reliable economic model could be used in a more appropriate scenario.
While macroscopic judgments exist, engineering questions arise: if the future is not a supermodel, how can there be collaboration between multiple models and who decides when to use whom?
The small model does it first, and then the big model calls for help.
PyroDash is defined as a cost-perceived, token-level model synchronous reasoning paradigm in which the entire system consists of three components: a trainable small model, a frozen large model, and a synchronous engine responsible for executing the switches (collaborate engine, abbreviated CE).
PyroDash collaborative reasoning architecture
The way it works can be summarized as: Small models prioritize, as many as once, one-way non-return.
When a request comes in, the CE first gives it a fixed system prompt to the small model, which starts generating tokens in a streamlined manner and continues to spit out tokens.
If it can be done independently, the result directly returns to the user, and the whole process does not produce any large model calls.
If at some point it judges itself to be unsuccessful, it issues an exclusive control token. After detecting this signal, the CE immediately stops decoding the small model, strips the control token, and once packs the original problem and parts of the previously formed reasoning trajectory into a frozen large model. The latter writes the answer along the trajectory. The control is no longer returned to the small model, and a request only generates one large model API call at most.
The paper gives an example. GSM8K asks a question: a seedling costs $90, binds seven oak trees a year, sells for $1.5 each, fertilizes for $3 a year, and after how many years does it start making money? The small model does not seek immediate help, but first simplifies it into an annual net income calculation, 7×1.5−3=7.5 dollars, and then sends out τ_off. What the big model sees when it takes over is no longer a natural language question, an already calculated annual net income, which is listed as 7.5n > 90, solving n > 12, pointing out that the 12th year is just back in the book, and only the 13th year began to earn.
There are several other examples in the paper's appendix. For example, AIME 2024's set of pairs of equations, in which the small model first converts the three pairs of constraints into three linear equations and rewrites the target into 4a + 3b + 2c to simplify the intersection; Minerva's one material science problem, in which the small model gives the density formula of the body's cubic crystals and indicates that the target value can be changed by converting the unit into a large model.
These tracks indicate that interconnection is not the same as abandonment. The work that a small model has done will be used as a rewrite of the larger model in the previous context, rather than being thrown back.
For businesses, the architecture has a more practical nature. Because the synchronous engine spells out the context, it doesn't require much of the large model. Kevin Ding also said on the podcast: "Open source, closed source can be, and we don't need its gradient". The large model here is just a rewrite endpoint, with no overtraining and no exposing of internal logits.
Why not pick a good model before you start?
Putting decision-making into the production process is the biggest difference between PyroDash and mainstream industry practices.
Speaking of size model collaboration, routing is often thought of first, such as FrugalGPT organizing multiple APIs into hierarchies, RouteLLM training a matrix decomposition router with preferred data, and Hybrid LLM allocating requests according to predicted difficulty. The common feature of this type of approach is that the decision takes place before decoding begins: read analysis input→ judging difficulty→ routing requests to a model.
However, this strategy is also problematic: the difficulty of real tasks is often not written in the input. A query may start with just extracting information and require complex judgments in the middle; the query level routing allocates the entire query before decoding, after which there is no room for adjustment of the system regardless of any new information that appears in the generation process.
A more sophisticated idea is to move decision-making into the generation process, and academics have come up with several strategies over the past two years: for example, CITER's strategy is to train a cost-sensitive MLP router to decide by which token to generate; Co-LLM is to make alternate decoding a hidden model choice; RelayLLM in January this year is to have the small model initially issue instructions requesting the large model to generate a boundary-length piece of content and then hand over control back to the small model; Glimp Router is completely free from training, with each logging step using the first token as an uncertainty signal and repeatedly calling the large model when necessary.
RelayLLM framework diagrams
The granularity is finer, and the price follows. The token-by-token evaluation router means that there is an extra expense for each step; the repeated switching request service cycle simultaneously coordinates the two models and maintains their decoding status; while under a non-state commercial API, each re-intervention involves repeating the growing context over and over again, with a few very short model extensions converting into several full prefill bills. In addition, Co-LLM has no explicit cost targets, and RelayLLM penalizes the share of large model tokens rather than the prefill and decoding prices in real accounts.
As for what looks like speculative decoding, it actually solves another problem. From the original schemes of people like Leviathan to SpecInfer, Medusa, MagicDec, this line optimizes delay and throughput, and large models score on each draft block, often requiring access to its output probability or even weight.
PyroDash is targeting these costs: decisions are completely internalized in small models, without the need for an out-of-the-box router; switching one-way and multiple times, without reversing; rewriting the interface with standard text only, without colliding with the interior of the model.
Write the bills in the reward function
Pyromind argues that it's about how small models know where to put their hands.
There is no τ_off token in the wording of the original small model, much less associate it with "the boundary of capability is here". The answer to PyroDash is a three-stage progressive pipeline that trains only small models.
PyroDash is a step-by-step training process in three phases.
Supporting the training is the Pyromind built EasyHard-24k data set, which contains samples of 24061 standard messages format.
Address of data collection:https://huggingface.co/datasets/pyromind/easyhard-24k
In the structure of the data set, it is difficult not to classify the static labels according to benchmark test results, but rather to define the small model that actually makes decisions in the future, which is to run through each sample using the basic small model, answering the same simple subset, answering the wrong but able to reconstruct the correct logical trajectory. This "model's relative difficulty" is more in line with synchronous reasoning than static labels, because easy samples mark the current small model that can complete the task independently, and hard samples mark the candidate that really needs help.
There is also a trap in the hard sample: the basic small model on these problems is the original trajectory itself that is wrong, and is taken directly as an SFT target, which is equivalent to retraining the path that led to the failure back into the model.
Pyromind's practice is to reconstruct a simple chain of thought that leads to the correct answer with a large model, and then dynamically insert one to four candidate intersections into the correct path. At the same time, the data is expanded into two prompts: one without synchronous prompts; no τ_off in the path to maintain the original independent inference behavior of the small model; and another with the addition of synchronous prompts, in which easy targets still do not contain τ_off capabilities, so as to both help open the model and not encourage it to do well.
EasyHard-24k data collection and statistics
The first stage is to learn to embed a control token. In order to avoid the instability caused by a completely random initialization, the input and output embedments ofτ_off are initialized into a set of focus point tokens embedded with an even-valued cactus noise, where the focus point is selected by symbols, alternating lines, terminators that represent natural loops, the noise standard being 0.1. This design allows the new token to be placed in the embedded space "near the border", in a sense it would have been a stop.
The second stage is behavior-cold-starting, with a round of SFTs mixed in two contexts, allowing the same model to learn two patterns simultaneously: the usual independent reasoning when there is no synchronous prompt, synchronous prompting and allowing switching signals to be sent when the subject exceeds capacity. By this time the model will have "expressed switching", but those inserts are just the starting point of behavior, not the best position in terms of cost and accuracy.
PyroDash uses cost-conscious GRPO to coordinate strategies and execute exactly the same one-way switching path as the reasoning: once τ_off occurs in a rollout, the training environment reproduces the packaging, terminates, calls freezes the large model, assembles the entire process of the joint output, and then rewards for the correctness of the final answer, as well as the cost of the joint path.
The reward is in the form of an accuracy subtraction of λ times the cost of unification, where the cost of unification is the absolute expense of this collaborative reasoning divided by the baseline expense of "the same problem completely given to the big model".
Key costs: It includes not only the tokens generated by the small model itself, but also the cost of entering the large model as a prefill, as well as the cost of decoding the large model afterwards.
That is, every token that a small model writes multiple times before the transfer must be counted twice on the bill: once by small model billing, and once by subtext billing on the input of a large model. This detail explicitly lists the part of the expense that the previous token covers up as an indicator of the proportion of the class into the training goal. When the integration cost is less than 1, the explanation is that synchronization is cheaper than a pure large model; greater than 1 indicates that redundant generation or ineffective transfer instead raises the bill, a trajectory that leads to a loss in the relative advantage within the group.
λ is an adjustable parameter: the smaller the system, the more willing it is to change the accuracy of the larger model; the larger, the more emphasis is placed on low-pressure calls and costs.
As Kevin Ding puts it: "RL is not only used to enhance the capabilities of a model itself, but it can also be used to train the entire system on how to work and find the right balance between efficiency and cost".
What is the real effect?
In the experiment, Pyromind used Qwen 3.5-4B as a small model, GLM-5.2-FP8 as a large model, and evaluated it on five mathematical logic criteria. Details are as follows:
Cost conversion benchmark (USD/million Token): small model input 0.05 output 0.08; large model input 0.90 output 2.86.
You can see that when λ takes different values in training, you get a completely different model of character, and the company can choose according to the value of the task and the budget.
On average, small models are called large models for every 100 questions, and the total estimated cost of the five benchmarks decreased from $49.36 to $1.78 for pure models, less than one-third of the original. The cost is only about 3 percentage points lower than the average accuracy of pure models.
The average accuracy of this grade is 64.04%, which is 6.36 percentage points higher than that of the pure GLM-5.2-FP8 at 57.68%, while the cost is also two percent lower.
Why would the accuracy rate reverse? Because at this point in time, almost every problem occurs when the supermassive model takes over, what is seen is not the original problem side, but the small model's well-arranged reasoning. This changes the state of it when it enters the problem. This explains: "let the small model learn to seek help" has added value beyond the cost advantage.
Look at λ = 0.1, its average accuracy is higher than RouteLLM and GlimpRouter, the two mainstream routing baselines, but costs only one tenth to one seventh of them. Both baselines hand over more than 75% of decoding tokens to large models, while PyroDash only hands out once when necessary. This also proves the effectiveness of the route to "move decisions into the generation process".
So, what is the contribution of GRPO to these gains? Pyromind also conducted a dissolution experiment.
The average accuracy stood at 46.25% only after the first two phases of supervision of the fine tuning; the accuracy jumped to 64.04% after the cost perception GRPO of the third phase. This shows that the cold start of the church is the behavior, and reinforcement learning is the judgment.
In summary, PyroDash is not a request-level routing in the traditional sense, but instead uses reinforcement learning to allow small models to learn when to communicate in the production process, thus making a dynamic trade-off between effect and cost. For developers, its appeal is to turn the decision-making process of "when to call a more expensive model" into a trainable, modifiable, reproducible model capability.
The second half of reasoning: from selecting models to improving organizational skills
PyroDash is the second route, but it doesn't deny the first.
"The better the baseline model, the less the pressure on the Worker Model". Kevin Ding believes that the architecture itself is a patch on the baseline model, and that baseline evolution is a good thing rather than a threat to it. He also says: "The production environment has a very realistic constraint, effect, cost and privacy that is difficult to solve simply by calling larger models simultaneously, some tasks are suitable for directly calling the baseline model, some are more suitable to be done on a smaller local model, and some require collaboration between the two. The stronger the baseline model, there is a need for more routing, synergy and cost optimization".
From a longer timeline, PyroDash may also change the definition of "reason".
Over the past two years, there have been three ways to optimize the logical side: reducing models, quantifying, distilling, speeding up decoding, speculative decoding, Paged Attention, and distributed routing.
The three paths have a common default: reasoning is one-off, atomized: a request comes in, is generated by a model from start to finish, which model the system chooses to use, and makes the model run faster and more economical.
PyroDash's work shakes this premise and includes three changes:
The position of the decision is moved from the beginning of the request to the generation process;
The decision-making entity becomes the model itself from an external router;
The goal of optimization is to move from token ratios, FLOPs, delaying these proxy indicators to real bills that are priced separately by prefill and decoding.
Combined, the three can conclude that reasoning can be continuously optimized through post-training and that costs can become part of training objectives.
Pyromind calls this a collaborative scaling paradigm: when a part of scaling's problem moves from "making a single model bigger" to "how to arrange heterogeneous model services", computing and intelligence can flow through a model network on demand.
We can also see data supporting this judgment: Local models below 100B on the Hugging Face have taken over the mainstream of downloads. Kevin Ding explains: "The world of demand is diverse, and it doesn't have to be all of the needs of a large Base Model to come up with a unified solution".
As for the next step, Kevin Ding says he wants PyroDash to gain the most mainstream recognition in the community, "to see if it really solves specific problems in the production scene, not just scores on Benchmark".
For more details, see Pyromind's official article: The cost of AI reasoning is down 96%, precision exceeds the basic model: PyroDash redefines AI reasoning in collaboration with multiple models.
PyroDash's papers, models, datasets, and evaluation code are open-source and officially recommended by the Morta community, and the model weight is based on Qwen 3.5-4B's PyroDash-4B-GRPO-Lambda-0.6, training data is 24061 EasyHard-24k, which developers can directly download and test on their own tasks.
Website of the project:
Open source code:https://github.com/Pyromind-Dynamics/PyroDash
Hugging Face：
What is the meaning of this word?