It's the first time in the history of the world.
Think of it this way:
You created a little app with Claude Code, and you gave it an AI suit.
He wanted it to automatically answer basic questions such as "how to log in" and "membership service has content" so that he could be a handyman.
In the end, the client can talk, but you become a full-time practitioner:
If you don't answer correctly, add a suggestion; explain too much and add a restriction.
What's even more frustrating is that every time you change a suggestion, you have to ask the usual question again, afraid that it will lose you, just fix a problem, but change the original answer to worse.
So the time saved by writing code with AI has been reduced to a round of artificial mediation.
On September 28, Anthropic officially released a correction guide to this type of repetitive debugging.
At the heart of this guide is this: repeatedly test, modify, and Claude can do it.
If you take a little bit more of it, you'll be less likely to be a bit of a hard-core tester and put more effort into turning on AI:
Give it a rule, tell it exactly what to do, and then check if it really does it.
"Don't talk blindly" becomes a specific test.In order for AI to change its direction, it has to be clear about "how to do right".
We always want AI to be "a little reliable", but it's too abstract.
If the requirements can be reduced to the problems that users will actually encounter, the criteria will be more specific.
For example:
When the user can't find the export button, can it give the correct click path?
When a user asks for a function that hasn't been developed yet, will it be fairly foolproof?
If there is a problem with a refund, can it be handled according to clear rules and not compromise on time?
The first official entry is:/claude-api build-eval.
Simply put, it can help you turn these specific requirements into test questions and build a library of receiving questions that can be run repeatedly for small applications that access Claude.
You can give it the usual question of how often AI flips a car, but the theme library can't just contain the "rolling scene", the most common, the most common questions are also covered.
Otherwise, there will be more puzzles and difficulties and it may not be possible to measure the everyday experience of ordinary users.
The subject is well chosen, and the "how to judge" should be made clear.
"Professional", "smart", "friendly", all of which require particulate matter to be too thick to be measured on a more specific scale.
For example, what information must be clear, what mistakes cannot be made, and to what extent it can be done.
Example of official mailbox classification: Test samples must be confirmed by the user.
Claude will help you come up with the corresponding rating rules, and you need to verify that the high-scoring answer is actually solving the user's problem.
Some results can be directly checked by the user, such as whether critical fields have been omitted; open questions that have multiple reasonable answers can be encountered, and models can be scored according to clear rules.
But the scores themselves have to be switched off first.
Take a few exaggerated answers and compare your own judgments to see if there is a "yes but no answer" or "no problem is solved but it's passed".
The results of the survey were verified and regularly checked.
For developers, the biggest thrill of this step is that you can finally turn repeated complaints of "how did you get it wrong" into a set of automated checks that can run continuously.
Then, every time I change a suggestion, I run again to see where it's been fixed or where it's gone wrong.
Changing your own measurementsIt's getting worse.The theme library was built and the second official entrance was opened: claude-api hillclimb.
Hillclimb can be understood as "climbing", allowing AI to try to optimize a bit and see if it makes any progress.
But the KPI also needs to be specific and to be clear about what it allows to change.
For example, let the customer answer a bit more accurately, allowing it to modify the prompts; or try to save the API fee on the assumption of maintaining the quality of the answer, allowing it to adjust the model configuration.
After receiving the assignment, Claude checks the erroneous questions used to iterate, makes a change each round, and then re-evaluates:
If there is an improvement under the established objective, the modification is retained; if the modification is completed, it is withdrawn.
According to official documents, both entrances require claude code v2.1.259 and above.
Tip words, skill documents, tool descriptions, and model configurations can all be optimized, but what you need to change specifically depends on your scope.
You might be worried that AI will become a test-tube teacher, or that you're just going to be stuck with the questions you've been asked, and you're going to start talking nonsense?
Officials have taken this into consideration.
The process leaves a portion of the question alone as an unpublished "acceptance question".
The models responsible for proposing the modifications did not see the contents of these questions. After each round of modifications, they were tested to see if the results really improved or only applied to the exercised questions.
If the test scores are low and the "acceptance test" score is not improved, there may be an "overfit": more and more tests will be done, but other tests will not progress.
In such a situation, Claude would withdraw the round of amendments; he would also withdraw if his performance had deteriorated after the change.
After each round of revisions, retention or withdrawal depends on the results.
This can help to identify the problem of "just doing the homework", but it doesn't mean that you can't worry about it.
The problem is that the book itself does not cover the real issues, and it can still make it roll over.
Official customer caseCost decreased to about 1/5Anthropic has demonstrated its effectiveness with an internal customer review.
On 14 unparticipated guideline modifications, the final configuration decision accuracy increased from 78.6% to 90.5%, an increase of 11.9 percentage points, and the cost of model calls decreased to about 1/5 of the original.
Adjusting the model, thinking about the intensity and prompts, improving accuracy and cost.
However, it's the result of a specific set of configurations, a specific subset of samples, that doesn't mean you can save 80% on this process.
The comparison of the model call costs cannot be taken as the total cost of the entire project.
But this is still attractive for individual developers who pay for their own APIs.
After all, cheap models do not have to be expensive if they always answer unanswered questions, forcing users to ask repeatedly, and finally you have to intervene manually.
A simple question can also be unnecessarily costly if it involves expensive configurations and produces an unnecessary amount of explanation.
Now, you can test answering quality and calling cost together, so that AI can try a more cost-effective solution that meets the quality requirements.
However, the optimization itself also consumes the cost of calling the model.
Decide how much you're willing to pay for the experiment, and then decide to let the AI run a few rounds to be more precise.
Give your time back to what really mattersOnce this set of automated processes runs, you get the chance to do less repetitive word-of-mouth fixes and put your energy back into the product itself.
The first is user insight.
You think it's natural that people who open an app for the first time can't even find the entrance. What you have to do is add these real-life puzzles to the library, and the next time AI optimizes, it's going to get more grounded.
The second is to set control standards.
Is it possible to say the same thing clearly, just to change the way things are said, that the raters will misjudge?
If the scoring criteria themselves are flawed, the AI is busy for half a day, the scores may have gone up, but the user's problem is still there.
Email Classification Assessment: Check the scores on the left, review the model input and response on the right, and check whether the score is valid.
And finally, it's the balance.
Do you miss out on the necessary steps? Will the pursuit of cheapness be at the expense of accuracy?
The idea of giving access to AI to applications is to be mindful.
If you end up changing your answer with it all the time, it's the other way around.
When Claude is able to do more repetitive debugging, ordinary developers can spend more time honing their little tools and small applications to get those idle ideas back on track.