Recently, the Jev model introduced by TypeSafe AI exploded, bringing the model form of "Input status → Output decision + probability" back into the public view. Developers only need to provide a state and question, and Jev can directly return pre-defined options, scores or event probabilities, and the results can be directly called by the code.
While Jev currently has no open source, architecture and training details are not disclosed, there are four core highlights from its public API and features:
✨ Direct decision-making + probability
⚡️ Support for parallel probability prediction
🎯 Probability itself is at the heart of the prediction, not just the attachment.
⚖️ Pursuing a probability calibration: not only choose yes, but let the given probability reflect true accuracy
These are key highlights, along with the NeurIPS 2025 Selection Paper.ConfTunerThe two are highly consistent in the technical intuition of probability modeling and calibration:It focuses not only on what the model ultimately produces, but also on the complete probability distribution behind the results of each candidate.
The Tokenized Brier Score method, proposed by a team at the National University of Singapore in ConfTuner, is a probability distribution that brings the model's "prep" closer to its actual accuracy by directly training candidate confidence.
Source: ConfTuner paper Figure 1. The paper uses medical scenarios to illustrate that models exhibit excessive confidence in incorrect answers that may affect subsequent judgments.
The paper:
The code is:https://github.com/liushiliushi/ConfTuner
The project is being implemented by JevTuner.
Putting the phrase "I'm 80% sure" into a set of token probabilities that you can practice.
Assuming the model answers a question, the next step is to output "Confidence: 80%". In this step of generating confidence, the model has calculated logits for the candidate token in the word table. ConfTuner extracts candidate tokens representing confidence from 0 to 100 and then softmax on these candidates to get the probability distribution.
This is a critical step: it's not just the "80%" that the subjects ultimately say, it's 0%, 1%... 100% the overall probability of each candidate's confidence matching.In the same calculation step, you don't have to repeat each number separately.。
Image source: ConfTuner paper Figure 2. The first half obtained the distribution of the candidate trust token, and the second half used the Tokenized Brier Score to fine-tune it.
This is where the two are very similar in technical terms. Jev's open interface gives developers the probability of getting a candidate decision at once; ConfTuner reads the probability of a candidate's trust from token logits at once and trains the entire distribution directly.How to generate and calibrate a set of probabilitiesIt's not just the end result.
How can you learn to "get it right" with just the wrong labels?
ConfTuner's core idea is to get the probability is just the first step.Tokenized Brier Score。
Setting
Confidence Token for model output
The probability of
For the numerical confidence represented by the Token,
Indicates whether the answer is correct ((true to 1, false to 0)). Its training loss function is defined as:
Intuitively, if a model answers a question incorrectly, assigning a higher probability to "99% sure" is severely punishable; on the contrary, if the answer is yes, the loss function encourages the model to produce a higher confidence. This greatly reduces the cost of data markup.
More importantly, the paper mathematically demonstrates that the loss function is a Proper Scoring Rule (proper scoring rule): predictions that minimize losses when expected for the same class of inputs necessarily focus the probability mass on the confidence token that is closest to the true correctness rate. This provides a solid theoretical guarantee for the model's probability calibration.
Experimental verification: A significant decrease in calibration errors
The research team trained on HotpotQA and performed generalized testing on multiple tasks such as GSM8K, TriviaQA, StrategyQA and TruthfulQA.ECE（Expected Calibration Error）The lower the difference between predicted confidence and actual accuracy, the better the calibration.
The paper report shows that in five datasets, the average ECE of the three base models of LLaMA, Qwen, and Ministral varies from0.2768、0.3781、0.4393It's coming down.0.1082、0.2872、0.1884：
Source:ConfTuner paper Table 1. The Average column on the right is the average of the five data sets referenced in the text; the lower the ECE, the better.
In the Reliability Diagram, ConfTuner demonstrates a significant reduction in the area of error between predictive confidence and actual accuracy, significantly better than other baseline methods.
Image source: ConfTuner paper Figure 3. The two rows are HotpotQA and TriviaQA respectively, with ConfTuner at the top right.
Training efficiency was also highlighted. Under the training conditions of the four A40 GPUs used in the paper, ConfTuner used 2,000 training data and completed the tweaking in about 4 minutes. Other training methods LACIE and SaySelf took 26 minutes and 120 minutes respectively.
Source:ConfTuner paper Table 6. Training time is based on 4 A40 GPUs, reasoning time is based on 1 A40 GPU; "Sample times" represents the number of responses generated by each input.
In addition, calibrated probability can also directly empower downstream tasks: for example, allowing low-confidence responses to trigger self-correction of the model, or routing it to a more powerful model processing. Under the same computational budget, model-leveling strategies based on ConfTuner have been improved by up to 9.3% and 5.5% on HotpotQA and TruthfulQA, respectively.
That's where the value of the Jev interface comes from.When probability directly enters code logic and determines the direction of a task, the probability calibration goes from a mere academic indicator to a cornerstone of the reliability of the entire system.
From trust tokens to decision tokens
Jev's red flag has also sparked a new thought: If the candidate Token no longer represents "0% to 100% certainty", but represents specific business decisions, can we continue to follow the idea of "direct training probability distribution"?
The ConfTuner team recently launched an open source extension project.JevTunerIt sets up single Token slots for candidate decisions and directly reads the Logits in those slots to obtain a complete decision probability distribution; during the training phase, it optimizes the probability using the correct decision labels and Brier Loss multiple classes.
The reasoning output format provided by the project allows developers to directly analyze the decision and probability distribution of each candidate as follows:
{"decision": "billing","probabilities": {"billing": 0.84,"technical": 0.09,"sales": 0.03,"other": 0.04}}
It should be noted that TypeSafe AI has not yet revealed the underlying architecture of Jev, and JevTuner does not claim to be an internal implementation of Jev, but provides a feasibility exploration along a technological path.Use candidate probability as the direct first output of the model and be truly responsible for these values themselves in training.
As more and more AI decisions are accessed by automated workflows, the "right choice" is merely linear; whether or not the model's given probabilities pass rigorous testing really determines whether these AI judgments can be trusted and used by the industry at scale.