Stop Vibe-Checking AI Agents: Build Tests That Measure Real Work

Stop Vibe-Checking AI Agents: Build Tests That Measure Real Work

AI agents can look impressive in a short demonstration and still fail in real work. An agent may produce a convincing answer while using the wrong tool, sending incomplete data, repeating an action, or missing the user’s goal. To make agents more useful, replace “it seems to work” testing with a small, repeatable evaluation system.

Agent tests need real tasks

An evaluation, often called an eval, is a structured test that gives an AI system a defined input and measures its result. Because AI systems can produce different outputs from the same input, an agent evaluation must measure more than the wording of the final response. It must also show whether the system completed the intended task and behaved safely.

OpenAI recommends task-specific evaluations that reflect real-world use rather than broad model metrics alone. Its guidance warns against “vibe-based” testing, in which a team decides that an agent appears reliable after inspecting a few outputs. A useful test set should represent the work the agent is expected to perform, including normal requests, incomplete information, ambiguous instructions, and likely failure cases. (OpenAI’s evaluation guidance)

Measure the entire agent trace

Once tests reflect real tasks, they should examine how the agent reached its result. The final answer is only one part of an agent’s performance. A stronger evaluation reviews the full trace: the tools called, the arguments sent to those tools, intermediate results, and the final outcome.

For example, a customer-support agent might receive full credit only when it identifies the correct policy, retrieves the right account record, applies the correct eligibility rule, and produces an accurate reply. A polished response should not pass if the agent accessed the wrong customer or invented a policy. Anthropic describes this approach as combining code-based checks, model-based grading, and human review. Its examples include verifying tool calls, parameters, outcomes, and the agent’s transcript. (Anthropic’s agent-evaluation research)

In practice, a useful evaluation scorecard usually includes four separate measures:

  • Task success: Did the agent achieve the requested result?
  • Tool accuracy: Did it choose the right tool and provide valid arguments?
  • Process quality: Did it avoid unnecessary loops, unsafe actions, and wasted calls?
  • Answer quality: Was the final response correct, complete, and understandable?

Use layered grading methods

These measures require different kinds of checks. No single grader is reliable for every type of agent behavior. Deterministic checks work well when the expected result can be compared with a known value. They can verify a database record, a file change, a status transition, a required field, or the parameters sent to an API.

Model-based graders can assess qualities that are harder to express with fixed rules, such as clarity, relevance, or whether a response followed a policy. Teams should calibrate those graders against examples reviewed by people. Human review remains important for high-impact tasks, unusual failures, and cases where an automated grader may reward a plausible but incorrect answer.

Google’s guidance makes a similar point: agents can loop, consume excessive tokens, or hallucinate while still returning a polite response. Testing must therefore look beyond the final text and inspect the agent’s actions and results. (Google’s agent-evaluation guidance)

Turn failures into regression tests

Layered checks become more valuable when teams use them to capture real failures. When an agent makes a mistake, preserve the original request, relevant context, tool trace, expected behavior, and actual result. Add that case to the evaluation set before changing the system. This helps prevent a fix for one problem from silently creating another.

Run the evaluation set whenever the model, system instructions, tools, retrieval sources, permissions, or business rules change. Track results over time instead of treating one score as a permanent quality label. OpenAI recommends logging development activity so teams can mine logs for useful test cases. Anthropic says evaluations become more valuable as an agent evolves. (OpenAI; Anthropic)

Set release gates for risk

The right release standard depends on what an agent can do. A read-only research assistant can tolerate a different failure rate than an agent that changes records, contacts customers, approves expenses, or writes production code. As an agent gains access and authority, teams should test edge cases, tool permissions, reversibility, and human-approval boundaries more carefully.

NIST’s draft guidance on AI benchmark evaluations stresses clear measurement objectives, documented evaluation procedures, result tracking, statistical analysis, and qualified reporting. For teams building agents, this means documenting what each test measures, keeping test data stable enough for comparison, and recording the conditions under which each result was produced. (NIST’s evaluation practices document)

A practical release gate can require critical tests to pass, tool-use errors to remain below a defined limit, and high-risk actions to receive human approval. The exact thresholds depend on the workflow. The central principle is simple: an agent should earn more autonomy by demonstrating reliable performance, not merely by producing impressive demonstrations.

References

  • Anthropic. (2026, January 9). Demystifying evals for AI agents. https://www.anthropic.com/research/demystifying-evals-for-ai-agents
  • National Institute of Standards and Technology. (2026, January). Practices for automated benchmark evaluations of language models (NIST AI 800-2 initial public draft). https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.800-2.ipd.pdf
  • OpenAI. (n.d.). Evaluation best practices. OpenAI Developers. https://developers.openai.com/api/docs/guides/evaluation-best-practices
  • Selbie, H., Zamora, D., Carah, J., & DeMartino, J. (2026, October 15). Evaluating AI agents: Moving past the vibe check. Google Developers Blog. https://developers.googleblog.com/en/evaluating-ai-agents-moving-past-the-vibe-check/