Evals in Practice: Agents, Retail, and Multimodal Pipelines

In Part 1, we covered what evals are, why agents need them, the three scorer types (deterministic, model-graded, human), and how to build an eval suite with a 35-line Python harness. This post applies those patterns to real problems: agent trajectory evaluation, retail catalog enrichment, customer support resolution, and multimodal pipelines.

Contents

• Agent evals: trajectory and tool-use
• Catalog enrichment: attribute extraction
• Customer support: resolution accuracy
• Beyond text: evaluating other modalities
• Common pitfalls
• The flywheel

Agent evals: trajectory and tool-use

For agents, output accuracy isn't enough. You also need to evaluate the path the agent took: which tools it called, in what order, and whether it recovered from errors.

Trajectory evaluation

A trajectory eval compares the expected sequence of tool calls against what the agent actually did:

The scoring considers three dimensions:

1. Correct steps: did the agent make the right tool calls?
2. Correct order: were dependent steps executed in the right sequence?
3. Extra steps: did the agent do unnecessary work (wasteful but not wrong)?

def trajectory_score(expected_steps, actual_steps):
    """Score an agent's trajectory against expected tool calls."""
    expected_set = set(expected_steps)
    actual_set = set(actual_steps)

    # Correct steps (order-independent for now)
    correct = expected_set & actual_set
    missing = expected_set - actual_set
    extra = actual_set - expected_set

    # Order score: of the correct steps, were they in the right order?
    correct_in_actual = [s for s in actual_steps if s in correct]
    correct_in_expected = [s for s in expected_steps if s in correct]
    order_correct = correct_in_actual == correct_in_expected

    return {
        "step_recall": len(correct) / len(expected_set) if expected_set else 1.0,
        "step_precision": len(correct) / len(actual_set) if actual_set else 1.0,
        "order_correct": 1.0 if order_correct else 0.0,
        "missing": list(missing),
        "extra": list(extra),
    }

Worked example: retail support agent

Consider a customer support agent that handles return requests. The customer says: "I received a damaged laptop and I want a refund."

Expected trajectory:

1. lookup_order(customer_id) : find the relevant order
2. check_return_window(order_id) : verify the order is within the return period
3. verify_damage_policy(order_id, reason="damaged") : check if damage qualifies
4. process_refund(order_id, type="full") : issue the refund

Eval dataset entry:

{
    "input": "I received a damaged laptop and I want a refund",
    "context": {"customer_id": "C-4821", "order_id": "ORD-99012"},
    "expected_trajectory": [
        "lookup_order",
        "check_return_window",
        "verify_damage_policy",
        "process_refund"
    ],
    "expected_resolution": "full_refund",
    "tags": ["returns", "damage", "electronics"],
}

Composite scorer for the agent (using exact_match from Part 1 and trajectory_score from above):

scorers = {
    "resolution": lambda ex, actual: exact_match(
        ex["expected_resolution"], actual["resolution"]
    ),
    "trajectory_recall": lambda ex, actual: trajectory_score(
        ex["expected_trajectory"], actual["tool_calls"]
    )["step_recall"],
    "trajectory_order": lambda ex, actual: trajectory_score(
        ex["expected_trajectory"], actual["tool_calls"]
    )["order_correct"],
}

This gives you three independent signals: did the agent reach the right answer, did it take the right steps, and did it take them in the right order? An agent that processes a refund without checking the return window gets full marks on resolution but fails on trajectory. That matters because it means the agent will approve refunds it shouldn't on other inputs.

Catalog enrichment: attribute extraction

The task: given a product description (and optionally an image), extract structured attributes like material, color, fit, pattern, and category.

Why this is hard to eval: some attributes are deterministic (category is either correct or not), some are fuzzy (is "navy" the same as "dark blue"?), and some are subjective (is the style "casual" or "smart casual"?). A single scorer can't handle all three.

The combination approach:

# Category: exact match against taxonomy
def category_scorer(expected, actual):
    return 1.0 if actual["category"] == expected["category"] else 0.0

# Color: hierarchy-aware fuzzy match
COLOR_HIERARCHY = {
    "navy blue": "blue", "sky blue": "blue", "royal blue": "blue",
    "maroon": "red", "crimson": "red", "scarlet": "red",
    "olive": "green", "lime": "green", "forest green": "green",
}

def color_scorer(expected, actual):
    exp = expected["color"].lower()
    act = actual["color"].lower()
    if act == exp:
        return 1.0  # Exact match
    if COLOR_HIERARCHY.get(act) == exp or COLOR_HIERARCHY.get(exp) == act:
        return 0.8  # Hierarchically correct (navy blue for blue)
    exp_parent = COLOR_HIERARCHY.get(exp)
    act_parent = COLOR_HIERARCHY.get(act)
    if act_parent and exp_parent and act_parent == exp_parent:
        return 0.6  # Same parent (navy blue vs royal blue)
    return 0.0

# Style/occasion: LLM-as-judge
def style_scorer(expected, actual):
    return llm_judge(
        question=f"Product: {expected['description']}",
        expected=expected["style"],
        actual=actual["style"],
        rubric="""Score 1-5:
5: Identical meaning
4: Correct but different wording (casual vs everyday)
3: Partially correct (missed one aspect)
2: Related but wrong (formal vs casual)
1: Completely wrong"""
    )

Sample eval run output:

category:     94.2% (471/500 passed)
  topwear:    97.1% (n=140)
  bottomwear: 95.0% (n=120)
  footwear:   91.3% (n=80)
  accessories: 88.5% (n=60)

color:        89.6% (448/500 passed)
  solid:      95.2% (n=310)
  pattern:    78.4% (n=190)    ← patterns harder to eval

style:        82.4% (412/500 passed)
  casual:     88.1% (n=200)
  formal:     84.2% (n=120)
  ethnic:     71.6% (n=80)    ← needs rubric refinement

The tag breakdown tells you where to focus: pattern-based colors and ethnic wear styles need work. Without the breakdown, you'd see 89% accuracy and think everything was fine.

Customer support: resolution accuracy

The task: given a customer inquiry, resolve it correctly. Did the agent reach the right outcome?

Eval dataset: built from historical support tickets where human agents recorded the resolution. Each ticket becomes an eval example:

{
    "input": "I want to cancel my order, it hasn't shipped yet",
    "context": {"order_id": "ORD-44123", "status": "processing"},
    "expected_resolution": "order_cancelled",
    "expected_actions": ["check_order_status", "cancel_order", "confirm_cancellation"],
    "tags": ["cancellation", "pre_shipment"],
}

Scorers:

scorers = {
    # Binary: did the agent reach the correct resolution?
    "resolution": lambda ex, actual: exact_match(
        ex["expected_resolution"], actual["resolution"]
    ),
    # Trajectory: did it follow the right process?
    "process": lambda ex, actual: trajectory_score(
        ex["expected_actions"], actual["tool_calls"]
    )["step_recall"],
    # Safety: did it avoid prohibited actions?
    "safety": lambda ex, actual: 1.0 if not any(
        a in actual["tool_calls"]
        for a in ["override_policy", "manual_credit"]
    ) else 0.0,
}

The safety scorer is worth highlighting. It checks that the agent didn't take actions it shouldn't have, like overriding company policy or issuing manual credits. An agent can get the resolution right but still fail the safety eval. In customer support, how you reach the answer matters as much as the answer itself.

Beyond text: evaluating other modalities

Vision: catalog image evaluation

When product images feed into your pipeline, you need evals at the image understanding stage. Two common tasks:

Attribute extraction from images: the model looks at a product photo and extracts color, pattern, sleeve length, neckline. The eval compares against human-labeled ground truth using the same combination scoring from the catalog enrichment section: exact match for binary attributes (has_collar: yes/no), fuzzy for continuous ones (color), LLM-as-judge for subjective ones (is this "bohemian" style?).

Image quality scoring: does the product image meet listing standards? White background, centered product, adequate resolution, no watermarks. These are deterministic checks that a vision model can grade:

image_quality_rubric = {
    "background": "Is the background white or transparent? (yes/no)",
    "centered": "Is the product centered in frame? (yes/no)",
    "resolution": "Is the image at least 1000px on the longest side? (yes/no)",
    "watermark": "Does the image contain watermarks or text overlays? (yes/no)",
}

Structured output: schema conformance

When your agent produces JSON (API responses, catalog records, structured reports), schema validation is the first line of defense:

def structured_output_eval(schema, actual, ground_truth=None):
    scores = {}

    # Level 1: Does it parse as valid JSON?
    try:
        parsed = json.loads(actual)
    except json.JSONDecodeError:
        return {"valid_json": 0.0, "schema_valid": 0.0, "field_accuracy": 0.0}
    scores["valid_json"] = 1.0

    # Level 2: Does it conform to the schema?
    try:
        jsonschema.validate(parsed, schema)
        scores["schema_valid"] = 1.0
    except jsonschema.ValidationError:
        scores["schema_valid"] = 0.0

    # Level 3: Are the values correct? (if ground truth provided)
    if ground_truth:
        correct = sum(1 for k in ground_truth if parsed.get(k) == ground_truth[k])
        scores["field_accuracy"] = correct / len(ground_truth)

    return scores

Three levels of strictness: can it produce JSON at all, does the JSON match the schema, and are the actual values correct? Most teams only check level 1 and miss schema drift or field-level regressions.

End-to-end vs per-stage evals

Multimodal pipelines need both types. Per-stage evals isolate where failures happen (the vision model misidentified the color vs. the JSON serialization dropped a field). End-to-end evals catch interaction effects (each stage is fine individually but the pipeline produces wrong results).

Run per-stage evals during development to debug. Run end-to-end evals as release gates.

Common pitfalls

Eval set contamination. If your eval examples leak into your prompts (as few-shot examples, as system prompt context, or into fine-tuning data), your eval scores are meaningless. Keep eval data separate and never use it in the model's input.

Overfitting to your eval. If you tweak your prompt until it scores 98% on your eval set, you've probably overfit. The prompt works for those specific examples but fails on production inputs that differ. Fix: hold out 20% of your eval dataset and never look at it during development. Only run it before release.

Vanity metrics. 95% accuracy sounds impressive until you realize 80% of your test cases are trivially easy. Report accuracy by category and difficulty level. The number that matters is accuracy on the hard cases.

Testing the happy path only. Your eval dataset has 400 examples of straightforward queries and 10 examples of edge cases. The aggregate score looks great, but your system falls apart on the inputs that actually cause problems. Deliberately overweight edge cases in your dataset.

Not versioning your eval datasets. You add 50 new examples and your score drops from 92% to 88%. Is the model worse or did you add harder examples? Without versioned datasets, you can't tell.

The flywheel

Evals aren't a testing practice. They're a product development practice.

The team that writes evals first ships faster. This sounds counterintuitive: writing evals takes time, and you could spend that time building features instead. But every hour spent on evals saves multiples in debugging, regression hunting, and "I think it got worse but I'm not sure" conversations.

The flywheel works like this:

1. Build an eval suite for your current system (even a small one, 30 examples)
2. Measure your baseline
3. Make a change (new prompt, new model, new tool)
4. Run the eval
5. If the score improved, ship it. If not, iterate or revert.
6. Add any production failures to the eval dataset
7. Repeat

Every cycle through this loop makes your eval suite better (more coverage, harder examples) and your system better (each change is validated). The compound effect is dramatic: after a few months, you have a comprehensive eval suite that catches regressions before they reach production, and a system that's been iteratively improved against real-world failure modes.

That's the answer to the question that silences the room. "Can you show me how you're measuring this?" Yes. Here's the eval suite. Here are the scores. Here's how they've improved over time. Here's what we're working on next.

---

The examples in this post use retail (catalog enrichment and customer support) but the patterns are domain-agnostic. If your system takes an input and produces an output, you can eval it. Start with 30 examples and a simple scorer. You can always add complexity later. Start with Part 1 if you haven't read it yet.