It's the first time in the history of the world.
In areas such as scientific simulation, financial computing, graphing algorithms, and operational optimization, a common problem is not "no program", but a program that is already running too slowly.
Giving it to a GPU is far more than just changing a programming language. To evaluate whether a large model can handle this task, the research team proposed AccelEval: starting from a credible CPU program, testing whether AI can generate the right results, end-to-end faster GPU implementation.
The benchmark covers six areas, 42 computing tasks and three input scales, and the system compiles 43 CUDA optimization strategies that will further extend the assessment to "why fast, and if these experiences can be replicated".
The code link:https://github.com/Echoscd/AccelEval
CPU is like a PhD student.GPUs are like thousands of elementary school students.
Figure 1: Explain the comparison between "a few PhD students" and "a few thousand elementary students".CPUwithGPUThe way it works.
Imagine two teams of solvers: a few PhD students on one side and thousands of undergraduates on the other. The former is suitable for handling complex, flexible judgments; the advantage of the latter is that a large number of members can complete rules at the same time.
In high-performance computing, GPU's advantages come from large parallels, such as the large number of "primary students", whereas the CPU is good at handling branching-intensive tasks, such as highly trained "PhD students".
Assuming that there are a million different arithmetic problems, and that the problems are shared by thousands of people at the same time, it is attractive, of course. But the actual process is often not that simple. Some problems have to wait for the previous answers to start; some people will modify the same results at the same time; and some tasks, questions, profiles, and answers take longer than the real ones.
Facing a large computing task, which computations can be done at the same time, how to allocate CPU and GPU tasks, who needs to wait for whom, where the data is stored, and how to combine multiple results securely, is a very complex and tedious question.
For example, calculating the shortest route. One way to find the shortest distance from one starting point to another is to repeatedly check each route to see if it leads to a shorter route. If the distance obtained in a round is fixed, each route can simultaneously calculate its own candidate results.
However, the candidate results to the same location must take the minimum value, and the next round must continue after this round is completed. Graph data should also be kept on the GPU as much as possible to avoid moving back and forth each round.
At this point, AI is not just "translating code into CUDA", it's reading computational dependence, finding parallel structures, and writing a suite of executable partitioning schemes.
It's 10 times faster.The whole process may slow down.
Figure 2: Core computing is faster, but that doesn't meanEnd to endThe process is faster; data transfers and closing costs will also affect overall consumption.
Even if AI writes a fast GPU kernel, it doesn't mean users can get results sooner.
A simple hypothetical example illustrates the problem: the CPU takes 10 milliseconds to complete a task; after switching to a GPU, the core computation takes only 1 millisecond, but preparation and input and transfer takes 6 milliseconds, and final and output transfer takes 6 milliseconds.
Based on core computation alone, it appears to be 10 times faster; but users actually wait 13 milliseconds, which is slower than it used to be.[2]
This is why AccelEval puts "end-to-end" in a central position. Each task must be timed through a unified computing input that receives the input of the host terminal, returns the output of the host terminal, allocates the necessary memory during the call process, transmits data, GPU computation and cleansing. Evaluation also sets up an audit mechanism to check whether the work that should be timed is stored outside the timing range.
What users need is not a small piece of code that is faster, but the same task, from input to answer, a really short time.
We already have KernelBench.Why do we need AccelEval?
Currently, there are a number of benchmarks in the community related to GPU kernel generation and optimization, such as KernelBench, TritonBench, etc.
They are driving an important direction: allowing large models to automatically generate accurate and efficient GPUs based on PyTorch algorithms or deep learning computations.
However, these tasks have their own characteristics.
A large number of common deep learning algorithms, such as matrix multiplication, integration, convolution and attention, already have relatively mature parallel paradigms and long-term accumulated optimization experience.
As these algorithms and their high performance achievements appear widely in open-source frameworks, technology blogs, and code repositories, model performance on these types of tasks may also benefit from widespread exposure to already implemented patterns.
Thus, relying solely on these familiar computing patterns, it is difficult to fully answer another more general question: When a model encounters a program that does not already have a GPU template available, is it really able to read the computational structure and design reasonable parallel schemes on its own?
AccelEval is geared towards more realistic requirements and broader computing scenarios. Deep learning algorithms typically define clarity, structural integrity, high parallelism, and are naturally suitable for GPUs, so the associated parallel patterns and optimization experiences are relatively mature.
However, in a large number of practical applications, users do not have a well-defined deep learning algorithm in their hands, and often only one that can run correctly, but one that is increasingly slow as the problem grows. Such programs may include multi-layered cycles, conditional branches, complex data structures, and interphase computing dependencies.
In this case, GPU acceleration is no longer just about rewriting a local computation to CUDA, but about understanding the computational structure of the entire process: which parts can be parallel, which dependencies must be retained, where the data should be stored, which computations should be integrated, and how the CPU and GPU work.
AccelEval wants to evaluate whether the model can start from a reliable CPU and find the right parallel and acceleration solutions on its own when faced with this type of more open, heterogeneous program.
42 mission assessments
Figure 3: AccelEval's overall assessment process: fromCPUReference process starts and comparesGPUThe results were accurate and the statistics improved at an end-to-end speed.
AccelEval consists of 42 tasks covering six areas of high-performance computing, scientific simulation, graphing algorithms, time-space algorithms, financial computing and operational optimization. The tasks are based on open-source, existing computing kits and classical algorithms or research problems, extracting actual algorithmic CPU computing and unifying input and output interfaces.
These include path computation, particle simulation and option pricing, as well as multi-cycle inventory replenishment, network revenue management, Gittins index computation, and one-stage linear planning and solution operations.
Such a range extends the question from "can AI write some kind of familiar GPU algorithm" to "can AI choose to accelerate in the face of computational structures in different fields".
Each task provides a natural language description, a CPU reference implementation, a fixed interface, and three small, medium, and large-scale test inputs. The GPU version submitted by the model must first be compiled and run successfully and then compared to the CPU output under the same input; numerical-sensitive tasks use predefined error tolerance limits.
At the same time, testing on three scales allows the evaluation to see something else: a program shows parallel advantages only after the problem has grown, or it is started with a delay in transportation costs on small tasks.
"To be right" and "to run fast" are two skills.
The paper evaluated eight large models on the NVIDIA H200 and performed cross-hardware testing on the RTX 4090. Gemini 3.1 Pro achieved a geometrically average acceleration rate of 71.2 times relative to fixed single-thread CPU references on a medium-sized, accuracy-verified task with only one code generated per mission; 161.5 times on a large scale.
These results show that automatic CPU-to-GPU acceleration already has real potential, but cannot be detached from the measurement conditions: acceleration is also a unified single-thread implementation over statistical-only tasks, and CPU control is not a universal advantage over fully optimized multi-thread CPUs.
It is also worth noting that there are different capability scales between models. Some models can write for more tasks, but not necessarily make them run faster; some implementations have been tested, but still do not make full use of parallel computing resources. Just looking at "how many questions to answer" is not enough to characterize GPU acceleration.
The study therefore examines at the same time the correct coverage, end-to-end acceleration, and how far away a program is from the "best correct implementation of the measured model" in the task. The best implementation here is only the best results observed in the evaluation, not the best theories of hardware or algorithms.
Artificial reference realization also provides a useful reference: even in 11 tasks with artificial CUDA baselines, where the fastest correct solution is selected for each of the eight models, the relative rate of artificial reference realization is about 0.75 on the cross-task geometry average.
43 GPUs: Accelerated engineering experience
Figure 4: AccelEval will be validated byGPUThe code was broken down into 43 categories of optimization strategies and further developed a two-stage strategy migration experiment.
If the rankings were to end, we'd be able to figure out who's faster, but we wouldn't know what to do next.
CUDA's own clear feedback signal acceleration is well suited to accelerating Agent evolution by increasing the sample size. Assuming we have 1000 attempts and 1000 results on a single code and evaluate the numerical results "100ms" and "400ms" alone, it is difficult to know whether the model has actually found the right acceleration strategy and techniques.
So we should be more concerned: why is this code running 100ms and another code running 400ms? What kind of code structure is causing this gap?
Through long-term practice, human engineers gradually form the CUDA "engineering experience" to determine how different processes should be accelerated.
In contrast, LLM can accomplish much more than human experiments in a short period of time, but these experiments are often isolated from each other and lack cross-tasking connections and summaries.
As a result, it is difficult to translate a large number of test errors into accumulative, portable optimization capabilities.
AccelEval hopes to further develop a quantifiable framework that can be migrated between different missions, incorporating once-in-a-lifetime optimization attempts into the simulated engineering experience of human engineers.
To this end, the research team first established a catalogue of 43 CUDA optimization strategies based on existing knowledge of GPU optimization, covering memory access and replication, computational reconfiguration, parallel organization, load balancing and host-end coordination.
The study combined static code detection with large-scale model code analysis to analyze strategies for 231 validated "model and task" combinations, matching performance results with code adoption practices.
These strategies are not just a few compilation options.
For example, kernel fusion can combine continuous computing phases, reducing intermediate data read-write and multiple starts; shared memory blocks allow a set of threads to be duplicated and moved to the data on the chip; and adjusting the data layout, loop order, or algorithm selection can directly change the way a task works on a GPU.
Studies have observed that structural optimization, such as core fusion, splicing, and thread workload adjustment, is strongly associated with higher performance on the same task, while optimization of individual kernel internal details tends to be less profitable.
The value of this strategy directory is that it explicitly summarizes the optimization experience involved in one-off generation and experimentation: not only recording which implementation is faster, but also recording why it is faster, and how computation is organized.
What is the meaning of this verse?
Figure 5: The overall improvement of task-specific strategy guidance is 1.78× significantly higher than the overall recommendation of length matching (1.18×).
Over the past few years, big-language models have been constantly refreshing people's notions of how stupid machines can be: they can do mathematical problems, write code, prove theorems, and approach human experts for increasingly complex reasoning tasks.
But what's really more promising is perhaps not how many questions AI will be able to answer, but when those capabilities will be able to get out of the screen and into the real world production process.
If future models not only generate code, but also understand systems like engineers, discover bottlenecks, summarize experiences and continually optimize them, then the change brought about by AI will not just be "helping people complete tasks", but will begin to be directly involved in computing, engineering, and even the evolution of industrial systems themselves.
What AccelEval wants to explore is the next step: to make AI not only understand how the world works, but also become part of driving the way the world works.