The paper is titled: ClawBench: Can AI Agents Complete Everyday Online Tasks?
GitHub:
The main page of the project:
The article links to:https://arxiv.org/abs/2604.08523
Hugging Face:
If you feel that an AI agent is just a step away from helping you book tickets, fill out sales forms, and submit your resume, this study, published by the authors of MMLU-Pro, TIGER Lab, UBC NAIL Group, and UniPat AI, CMU, and others from the University of Waterloo, may calm you down.
ClawBench, a new evaluation framework that allows AI agents to perform their daily tasks on 144 real-world production sites, came up with disastrous results:
The most powerful of these is the Claude Sonnet 4.6, which takes two turns on each of the three missions. GPT-5.4 is only 6.5% tonne, and only 10 of the 153 missions have been completed.
It's not a question of whether AI will use a browser, it's a question of whether AI will use a browser.
It's the question of "Can AI work for you?"
The existing web agent ratings are mostly as follows:
WebArena: In my own sandbox, the website is fake, the page is static, what's wrong?
OSWorld: Running in a virtual machine, running on nine apps, going back and forth and testing those tricks.
Mind2Web: A more thoughtful, straightforward selection of operational trajectories.
Front-end models can easily score 65%-75% on these assessments, which seems to allow agents to execute real-world tasks on a timely basis.
But ClawBench does something different: it lets AI access real web pages directly from a browser -- not sandboxes, not simulators, but the production environments that people use every day, like Google Flights, DoorDash, Zillow, Instacart, Airbnb, LinkedIn, Doodle -- 144 platforms, 153 tasks, covering 15 daily categories from shopping, subscriptions, resumes to filling insurance policies.
In other words, it examines whether AI can act as a substitute for the user to accomplish the complex task that really needs to be done by changing the interaction and understanding the state of the website.
The result: The high scores in those sandboxes are almost clear on the real site.
The original text of the paper was bloody:
"Strong performance on existing web-agent benchmarks does not transfer to everyday write-heavy tasks on live production websites."
It translates to: You think you're a college student, and you're changing the exam directly into a white paper.
The problem of security: how to make AI easy?
And don't let it really spend the money?
There's a technical dilemma to this benchmark: you let the AI write on a real website, and it's going to make an order, it's going to post a resume, it's going to send an email, and who's behind it?
ClawBench's solution is ingenious, called final-request interception.
The practical approach is to: Human taggers go through each task in its entirety, marking which "irreversible last hit" is the POST request that corresponds to the "order" button on the HTTP request bar, such as the DoorDash. Then, as the AI agent executes the task, the system monitors all exit site traffic via Chrome extension and the CDP (Chrome DevTools Protocol).
In other words, AI can browse, search, fill out forms, and click on a real website as normal as a human user, but when it tries to send out that tagged terminal request, the system intercepts it directly and the request never reaches the server.
It's not "not smart enough".
It's the last mile of collective phobia.
The paper's most interesting finding lies in its detailed analysis of the failure trajectory.
On the surface, Claude Sonnet 4.6 failed because the artificial verification walls intercepted (Cloudflare, DataDome verification codes rejected automatic browsers directly), logged in, and filled in the wrong form fields. These were "hard wounds".
But there is a problem that occurs repeatedly in all models, which the paper calls "last-mile non-commitment" phobia.
What does it mean? The agent can do all of the above: find the right website, search for the right merchandise, click on the right page, fill out the right form... and then, when you need to click the "Submit"/"Confirm"/"Order" button, it stops.
In the paper, there is a case that makes people cry and laugh:
ClawBench Mission 780: Complete a purchase process on the Korean anime website Soko Glam. Claude Sonnet 4.6 perfectly finds the merchandise, adds it to the shopping cart, enters the settlement page, starts filling in the delivery address.
The paper statistics show that six of the eight models, with a large number of failed tracks, accumulated at the stage of "reaching the final confirmation page but not submitted".
Another surprising finding was that the number of browsers that lost trajectories was higher than the number of successful trajectories. Claude Sonnet 4.6's average failure rate was 120 times, compared to only 64 times. Gemini 3 Flash's average failure rate was 74 times, and 45 times.
What does this mean? AI isn't exploring enough, it's playing on the ground. When you hit a validation code, you refresh it repeatedly, when you hit a form that doesn't pass through a refill, when you're blocked by a counter-crawler wall, you're constantly retesting, and a lot of tokens and API fees burn in a dead cycle without any progress.
What's up with GPT-5.4?
The most surprising thing on this list is not who won, but who crashed.
GPT-5.4, the flagship model of OpenAI, took only 6.5% on ClawBench.
But what's more bizarre than this is its pattern of behavior:
On average, only 13 instrument calls are made per mission (Qwen 3.5 is 42 and Sonnet 4.6 is 61).
On 85 missions, the agent exited directly, saying, "I'm done", and then walking away.
The average running time of Sonnet 4.6 is only 196 seconds, while the average running time of Sonnet 4.6 is 593 seconds.
The paper's diagnosis was that "early termination" was not a lack of ability and was not taken seriously at all. GPT-5.4 seems to tend to be "quickly judging tasks too complex and then giving up", rather than insisting, like Sonnet 4.6, on either succeeding or breaking the wall.
And in the mission it really tries, it exposes another problem: false completion.
ClawBench Task 2: Pointing a Big Mac and a 20-bit Mac at the DoorDash. GPT-5.4 searches for a McDonald's, points to a Big Mac, adds a shopping cart, and then the Agent himself declares "Both items are correct", believing the task is complete.
In other words, GPT-5.4 not only doesn't like to work, but also makes mistakes that you don't even know you're doing.
Qwen 3.5: The decency of the open source model
In a gloomy fog, Qwen 3.5 is the only bright colour worth seeing with both eyes.
With a completion rate of 26.1% (second place), the API cost per task of only $1.02 tons is one-seventh of Sonnet 4.6. The token that consumes every successful percentage point is the lowest of all models, with an average of only 42 tool calls per task, and is the "most efficient player to play".
Even more difficult is the execution discipline demonstrated by Qwen 3.5. A typical example of this is the task of filling out an extended multi-page form on an insurance quote website called Insurify. The task has about 40 consecutive screen pages, which involve tiered dependent fields (e.g. the following options will change) Qwen 3.5.
In the same mission, GPT-5.4 0 operations left directly, while Gemini 3.1 Flash Lite was delayed after 9 operations.
Of course, 26.1% is far from "usable". But it shows one thing: in real-world agent missions, open-source models rely on executive discipline, and not model sizes, to compete on the same scale as closed-source flags.
GLM-5: The most money-saving player.
The GLM-5 is another data point worth considering.
With a completion rate of 24.2%, it ranks third overall, but the API cost per mission is only $0.64 ton, less than one tenth of Sonnet 4.6. In the cost-performance coordinate system, GLM-5 is the one closest to the origin.
However, the GLM-5 has a noticeable shortcut: 48.4% of missions ended due to overtime (a 30-minute time limit), the highest of the eight models. This indicates that it can play, but slowly: there is not enough time for long-range planning and ongoing operations.
Who is the king in what mission?
The answer is that no one can eat.
The above details the important information in the results, but ClawBench's scores, broken down by scenario, reveal another interesting finding: different models excel in completely different areas, and no one model can do it face-to-face.
Shopping(Shopping): Sonnet 4.6 is 62% overwhelming.
Social: Sonnet 4.6 is down 75%.
Work (work): GLM-5 climbed to the top with 38.1% and Sonnet came in third.
Dev & Tech (development technology): Haiku 4.5 and Gemini 3 Flash ranked first, with only 27.8% of the chips, but also twice as good as Sonnet's 11.1%.
Travel: Qwen 3.5 and Gemini 3 Flash are number one, while Sonnet is only number four.
Job Search: The army is overwhelmed, the best Haiku 4.5 is only 12.5%, Sonnet 0%.
Automation (automation): Sonnet 0%. This pattern of "East is East is West" illustrates: the current web agent is not a generic capability, but a bunch of niche-specific performance puzzles. A model that kills everyone on a shopping site may not move even the first step on a job search site.
Human vs. AI:
It's the same browser, a completely different game.
ClawBench also recorded a data dimension that was almost never taken seriously in other benchmarks: interactive dynamics. The paper compares human and AI agents' operational characteristics in the same set of Chromium browsers, and the gap is as large as two species:
Human typing is one-to-one typing, and AI agents are instantaneous infusions of text. Humans have mouse trajectories, and AI does pixel-level clicks and jumps. This "unlike human" behavior is precisely why many anti-crawler systems recognize bots at a glance.
The subtext of the paper is clear: AI agents fail on real websites, not only because they are "not smart enough" but also because "the action is too unlike human" and this is precisely the problem that sandbox benchmarks will never expose.
Anti-bot: The toughest wall in front of an AI agent
Of all the reasons for failure, the highest frequency of occurrence is not a technical problem, but an engineering reality: an anti-crawler system.
Cloudflare Turnstile, Perimeter X, DataDome... These sites used to identify and block machine traffic systems are the strongest line of defense against AI agents.
A typical example of this is the paper:
GLM-5 was asked to go to Zillow to search a downtown Toronto apartment and submit a rental application. It opened Zillow, encountering the Perimeter X's "Push to Confirm You're Human" dialog box. For the next 33 turns, it attempted to click on the tractor, DOM detection, and even inject synthetic touch events.
Ironically, GLM-5 accurately diagnoses the problem, and it clearly describes it as "a cloudflare or similar bot-protected page, a verification mechanism specifically designed to detect and block automated interactions". But knowing what the problem is and being able to solve it is a gap that it doesn't cross.
This happens repeatedly on ClawBench: AI can recognize "I've just been authenticated", but can't go through it. Compared to humans, these models lose a bit on the "entry qualification" of the real Internet.
ClawBench and ProgramBench:
Two paths, one conclusion.
If you think the title is familiar, yes, in May 2026, we reported on the SWE-Bench original workhorse release of ProgramBench: Let AI rebuild real software projects like FFmpeg, SQLite, ripgrep from scratch, resulting in Claude, GPT, Gemini all with 0% completion rate.
ClawBench and ProgramBench, in parallel, point to the same conclusion:
ProgramBench examines what "AI can do as a software engineer": designing and implementing a system from scratch.
ClawBench's test is "Can AI be your personal assistant": doing a daily thing for you on a real website.
Combined, the two draw a clear line: AI is already powerful at generating code and navigating web pages in controlled environments, but when faced with the complexities of the real world, it requires long-term planning, multi-step execution, response to accidents, and understanding of context.
This isn't a problem that "plus one year of data and computing can solve it". ProgramBench exposes a lack of system-level architectural capabilities, and ClawBench exposes a lack of real-world interactive robotics.
So how far are we from "AI working for me"?
ClawBench's 33.3% ceiling, which translates into everyday language, is like this: You let AI help you do one thing on each of the three sites, and it can do the whole thing on one of them. The other two things, either halfway out, or the verified code is stuck outside the door, or you fill out the form but don't dare to submit it, or you declare yourself "fucked up" but nothing actually happened, or you just walk out.
But from another perspective, three things can be done for one, which was a joke three years ago, is 33.3% today.
The real value of ClawBench is not in the ability of the agent to score a low score, but in the first systematic measurement of how far away the leap is from "AI from the lab to the real Internet". The five-layer track record in the paper, the Agent-as-Judge protocol, the final-request interception mechanism, constitutes a scale that can be measured repeatedly. The next time a model gets 50%, 70%, or even 90% on ClawBench, that's really exciting.
Author's note
Zhang Yu, PhD student at the University of British Columbia (Individual homepage: AI, Reinforcement Learning, Auto Research, Recursive Self-Improvement, LLMs, related results published in international conferences such as ICLR, COLM, EMNLP).
Dr. Wang Liubo, PhD student at the University of Waterloo (personal homepage: of LLMs, multimodal reasoning, evaluation, related results published in international conferences such as NeurIPS, ICLR, COLM, EMNLP, ACL).
