LLM Evaluation

/llm-evaluation

Comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing. The user describes their application and quality concerns (or pastes sample inputs/outputs and existing test cases); deliver an evaluation plan, metric selection with rationale, and runnable eval harness code as chat artifacts. For improving the prompts being evaluated, use prompt-engineering.

When to Use

• Measuring LLM application performance systematically
• Comparing different models or prompts
• Detecting performance regressions before deployment
• Validating improvements from prompt changes
• Establishing baselines and tracking progress over time
• Debugging unexpected model behavior

Core Evaluation Types

1. Automated Metrics

Fast, repeatable, scalable evaluation using computed scores.

Text Generation:

• BLEU: N-gram overlap (translation)
• ROUGE: Recall-oriented (summarization)
• METEOR: Semantic similarity
• BERTScore: Embedding-based similarity
• Perplexity: Language model confidence

Classification:

• Accuracy: Percentage correct
• Precision/Recall/F1: Class-specific performance
• Confusion Matrix: Error patterns
• AUC-ROC: Ranking quality

Retrieval (RAG):

• MRR: Mean Reciprocal Rank
• NDCG: Normalized Discounted Cumulative Gain
• Precision@K: Relevant in top K
• Recall@K: Coverage in top K

2. Human Evaluation

Manual assessment for quality aspects difficult to automate.

Dimensions: accuracy (factual correctness), coherence (logical flow), relevance (answers the question), fluency (natural language quality), safety (no harmful content), helpfulness (useful to the user).

3. LLM-as-Judge

Use stronger LLMs to evaluate weaker model outputs.

Approaches:

• Pointwise: Score individual responses
• Pairwise: Compare two responses
• Reference-based: Compare to gold standard
• Reference-free: Judge without ground truth

Quick Start

from dataclasses import dataclass
from typing import Callable
import numpy as np

@dataclass
class Metric:
    name: str
    fn: Callable

    @staticmethod
    def accuracy():
        return Metric("accuracy", calculate_accuracy)

    @staticmethod
    def bleu():
        return Metric("bleu", calculate_bleu)

    @staticmethod
    def bertscore():
        return Metric("bertscore", calculate_bertscore)

    @staticmethod
    def custom(name: str, fn: Callable):
        return Metric(name, fn)

class EvaluationSuite:
    def __init__(self, metrics: list[Metric]):
        self.metrics = metrics

    async def evaluate(self, model, test_cases: list[dict]) -> dict:
        results = {m.name: [] for m in self.metrics}

        for test in test_cases:
            prediction = await model.predict(test["input"])

            for metric in self.metrics:
                score = metric.fn(
                    prediction=prediction,
                    reference=test.get("expected"),
                    context=test.get("context")
                )
                results[metric.name].append(score)

        return {
            "metrics": {k: np.mean(v) for k, v in results.items()},
            "raw_scores": results
        }

# Usage
suite = EvaluationSuite([
    Metric.accuracy(),
    Metric.bleu(),
    Metric.bertscore(),
    Metric.custom("groundedness", check_groundedness)
])

test_cases = [
    {
        "input": "What is the capital of France?",
        "expected": "Paris",
        "context": "France is a country in Europe. Paris is its capital."
    },
]

results = await suite.evaluate(model=your_model, test_cases=test_cases)

Evaluation Plan Workflow

1. Clarify what "good" means for this application; pick 2-4 dimensions that matter.
2. Match each dimension to the cheapest reliable signal: automated metric, LLM-as-judge, or human review.
3. Build a test set from real traffic or the user's pasted examples; include hard and adversarial cases.
4. Establish a baseline run, then gate changes (prompt edits, model swaps) on non-regression.
5. Deliver the eval harness, judge prompts, and a results-reporting format as chat artifacts.

Detailed Patterns and Worked Examples

Judge prompt templates, pairwise comparison harnesses, statistical significance for A/B tests, and CI regression gating live in references/details.md. Read that file when the navigation tier above is insufficient.

Usage

/llm-evaluation $ARGUMENTS