Evals for AI Agents: What They Are and How to Build Them

Most AI teams focus on the wrong things. Here's a scene that plays out constantly:

AI TEAM: Here's our agent architecture. We've got RAG here, a router there, and we're using this new framework for...

ME: [holding up my hand] Can you show me how you're measuring if any of this actually works?

... Room goes quiet.

Teams invest weeks building complex AI systems but can't tell me if their changes are helping or hurting. With new tools and frameworks emerging weekly, it's natural to focus on tangible things we can control: which vector database, which model, which agent framework. But the teams who succeed barely talk about tools at all. They obsess over measurement and iteration.

This post is about the measurement part: evaluations, or evals. It covers what evals are, why agents make them critical, and how to build an eval suite from scratch. In Part 2, we apply these patterns to real agent and retail use-cases.

Contents

• What evals are (and aren't)
• Why agents make evals non-negotiable
• Anatomy of an eval
• Building your eval suite

What evals are (and aren't)

Evaluations test model outputs against criteria you define. You give the model an input, it produces an output, and a scoring function grades whether that output meets your expectations. That's the entire concept.

But evals are easy to confuse with things that look similar:

	Unit tests	Evals	Benchmarks	Monitoring
When	Build time	Build + release	Model selection	Production
Inputs	Fixed	Curated dataset	Standard dataset	Live traffic
Grading	Pass/fail	Scored (0-1, rubric)	Leaderboard rank	Alerts, dashboards
Purpose	"Does the code work?"	"Does the model work well enough?"	"Which model is better?"	"Is it still working?"

The key distinction: unit tests have deterministic expected outputs. Evals don't. When you ask a model to summarize a document, there's no single correct answer. There are better and worse answers, and the eval's job is to distinguish between them.

An eval is a contract with three parts:

Dataset: a curated set of inputs paired with expected outputs (or expected behaviors). These come from production logs, hand-labeled examples, and deliberately constructed edge cases.

Task: running the model (or agent) on each input and capturing the actual output.

Scorer: a function that grades the actual output against the expected output. This is where the design decisions live.

Why agents make evals non-negotiable

A single LLM call with 95% accuracy sounds great. But agents chain multiple calls together, and errors compound:

A customer support agent that identifies the customer, looks up the order, checks the return policy, verifies eligibility, and processes a refund is making five dependent decisions. If each step is 95% accurate independently, the end-to-end accuracy drops to 77%. At 90% per step, it's 59%.

This is why agents need evals more than single-call applications. Three specific reasons:

Model upgrades break things silently. When you upgrade from one model version to the next, there's no stack trace if the output gets worse. The agent still runs, still produces output, still looks like it's working. Only an eval catches the regression.

Prompt changes are invisible regressions. You tweak a system prompt to fix one edge case and break three others. Without evals, you won't know until users complain. With evals, you know before you merge.

The virtuous cycle depends on measurement. The optimization loop for any AI system looks like this: write evals, run the model, measure, improve the prompt, measure again. Without the measurement steps, you're flying blind. With them, every iteration makes the system better, and you can prove it.

Anatomy of an eval

Deterministic scorers

When the correct answer is unambiguous, use a deterministic scorer. These are fast, cheap, and reliable:

def exact_match(expected, actual):
    """Simplest scorer: does the output exactly match?"""
    return 1.0 if actual.strip() == expected.strip() else 0.0

def json_schema_valid(schema, actual):
    """Does the output conform to the expected JSON schema?"""
    try:
        jsonschema.validate(json.loads(actual), schema)
        return 1.0
    except (json.JSONDecodeError, jsonschema.ValidationError):
        return 0.0

def regex_match(pattern, actual):
    """Does the output match a regex pattern?"""
    return 1.0 if re.search(pattern, actual) else 0.0

These are building blocks. In the eval harness below, you wrap them in lambdas that extract the right fields from each example:

scorers = {
    "action": lambda ex, actual: exact_match(ex["expected_action"], actual["action"]),
    "format": lambda ex, actual: json_schema_valid(SCHEMA, actual["raw"]),
}

Use deterministic scorers for: category classification, structured output validation, format compliance, boolean decisions.

Model-graded scorers (LLM-as-judge)

When the correct answer is subjective or there are many valid phrasings, use a second LLM to grade the output:

def llm_judge(question, expected, actual, rubric):
    """Use a grader model to score the output against a rubric."""
    prompt = f"""Grade the following output on a scale of 1-5.

Question: {question}
Expected answer: {expected}
Actual answer: {actual}

Rubric:
{rubric}

Return only the numeric score."""

    response = call_model(prompt, model="claude-sonnet-4-6")
    return int(response.strip()) / 5.0

LLM-as-judge is powerful but has known failure modes. Watch for position bias (the judge favors whichever answer appears first), verbosity bias (longer answers score higher regardless of quality), and self-preference (a model grades its own outputs higher than a competitor's). Mitigations: randomize answer order, cap response length in the rubric, use a different model family as the judge than the one being evaluated.

Human evals

Some things only humans can grade: tone, brand voice, whether an explanation actually makes sense to a domain expert. Human evals are expensive and slow, so use them strategically:

• For initial dataset creation (label 200 examples, then use LLM-as-judge for the rest)
• For periodic calibration (does your LLM judge still agree with human judgment?)
• For high-stakes decisions (medical, legal, financial advice)

Choosing a scorer

Most real systems use composite scorers: different scoring methods for different dimensions of the same output. A catalog enrichment eval might use exact match for product category, fuzzy match for color, and LLM-as-judge for style description. The composite score is a weighted combination.

Building your eval suite

Curating eval datasets

The best eval datasets come from production. Not synthetic examples, not imagined edge cases: real inputs that your system actually encountered.

# Minimal eval dataset: list of dicts
eval_dataset = [
    {
        "input": "Customer says: I ordered a blue shirt but received red",
        "expected_action": "initiate_return",
        "expected_reason": "wrong_item_received",
        "tags": ["returns", "color_mismatch"],
    },
    {
        "input": "Where is my order #12345?",
        "expected_action": "track_order",
        "expected_reason": "status_inquiry",
        "tags": ["tracking", "simple"],
    },
    # ... 200+ examples covering edge cases
]

Three rules for dataset curation:

1. Include failures. Pull examples where your system got it wrong in production. These are your most valuable test cases.
2. Tag by category. Tags let you see accuracy broken down by use case, not just in aggregate. You might be 95% accurate overall but 60% accurate on return policy edge cases.
3. Version your datasets. When you add new examples, you need to compare against the same baseline. Git works fine for this.

A minimal eval harness

You don't need a framework. Here's a complete eval harness in Python:

import json
import time

def run_eval(dataset, task_fn, scorers, output_path="eval_results.json"):
    results = []
    for i, example in enumerate(dataset):
        # Run the task
        start = time.time()
        actual = task_fn(example["input"])
        latency = time.time() - start

        # Score each dimension
        scores = {}
        for name, scorer_fn in scorers.items():
            scores[name] = scorer_fn(example, actual)

        results.append({
            "id": i,
            "input": example["input"],
            "expected": example.get("expected_action"),
            "actual": actual,
            "scores": scores,
            "latency": latency,
            "tags": example.get("tags", []),
        })

    # Compute aggregates
    for scorer_name in scorers:
        values = [r["scores"][scorer_name] for r in results]
        avg = sum(values) / len(values)
        print(f"{scorer_name}: {avg:.2%} ({sum(1 for v in values if v >= 0.8)}/{len(values)} passed)")

        # Break down by tag
        tags = set(t for r in results for t in r["tags"])
        for tag in sorted(tags):
            tagged = [r["scores"][scorer_name] for r in results if tag in r["tags"]]
            print(f"  {tag}: {sum(tagged)/len(tagged):.2%} (n={len(tagged)})")

    with open(output_path, "w") as f:
        json.dump(results, f, indent=2)

    return results

That's 35 lines. It runs every example through your task function, scores each dimension, prints aggregate results broken down by tag, and saves the full results for later analysis. No dependencies beyond stdlib.

Evals in CI

Two eval tiers work well in practice:

Fast evals (every commit): Run ~30 hand-picked examples covering the most important cases. These finish in under 2 minutes and catch obvious regressions: broken prompts, wrong model ID, schema changes.

Full evals (before deploy): Run the complete dataset (~500+ examples) including edge cases, adversarial inputs, and cross-category coverage. These take 10-15 minutes and catch subtle regressions: degraded accuracy on a specific category, increased latency, style drift.

# GitHub Actions example
on:
  pull_request:
    paths: ["prompts/**", "src/agent/**"]

jobs:
  fast-eval:
    runs-on: ubuntu-latest
    steps:
      - run: python run_eval.py --dataset evals/fast.json --threshold 0.90
  full-eval:
    if: github.base_ref == 'main'
    runs-on: ubuntu-latest
    steps:
      - run: python run_eval.py --dataset evals/full.json --threshold 0.85

The threshold is the minimum average score required to pass. Set it based on your current baseline, not an aspirational target. If your system currently scores 87%, set the gate at 85% to catch regressions without blocking progress.

---

That covers the concepts and infrastructure. You now know what evals are, why agents need them, how to choose a scorer, and how to wire evals into CI. In Part 2: Evals in Practice, we apply these patterns to agent trajectory evaluation, catalog enrichment, customer support, and multimodal pipelines.