Evaluation Methodology

/evaluation-methodology

Design evaluation suites for agent skills, plugins, and skill catalogs: what to measure, how to grade it, and where to set the bar. The deliverable is a concrete eval design the user can implement -- dimension definitions, fixture sets, grader specs, thresholds, and a run plan. Judging a single skill draft by hand is skill-judge; evaluating an LLM application in production code is llm-evaluation.

The Three-Layer Architecture

Stack evaluation layers from cheap to expensive; each catches what the previous cannot. Most quality signal comes from running all three at different cadences, not from one perfect test.

Layer 1 -- Static checks (instant, deterministic, run on every change). Pure inspection of the skill text: frontmatter present and well-formed, description length and trigger-phrase presence, body length inside a sweet spot (roughly 200-600 lines; under 100 is a stub, over 800 without layering is bloat), directive density (an excess of MUST/NEVER/ALWAYS signals micromanagement), dead cross-references, duplicated content. Static checks cannot measure whether a skill is GOOD -- only whether it is malformed. Their job is gating the expensive layers.

Layer 2 -- LLM judge (seconds-to-minutes, non-deterministic, run on release candidates). A model scores dimensions that need reading comprehension: triggering accuracy, output quality on simulated tasks, scope calibration. Judges need anchored rubrics (below) and, when stakes are high, multiple judges with an agreement statistic -- report inter-judge agreement, and treat low agreement as a rubric bug, not noise to average away.

Layer 3 -- Behavioral simulation (expensive, statistical, run before publishing). Feed N real prompts through the actual routing/execution path and measure what happens: activation rate, output consistency (variance across runs), failure rate with proper confidence intervals, token cost distribution. Small N lies: 10 prompts cannot distinguish 70% from 90% activation. Aim for 50+ prompts per skill for a publishable score, and report intervals, not point estimates.

Choosing Dimensions and Weights

Weight by "value destroyed when this fails". A defensible default ordering:

1. Triggering accuracy (heaviest, ~25%) -- a skill that never fires, or fires on the wrong requests, has zero value regardless of content quality. Measure as F1: recall over should-trigger fixtures, precision over should-not-trigger fixtures.
2. Output quality (~15-20%) -- does following the skill produce correct, complete results on representative tasks?
3. Scope calibration (~10-15%) -- neither stub nor bloated monster; content depth matches the category.
4. Consistency/robustness (~10%) -- variance across runs and behavior on edge-case inputs.
5. Efficiency, structure, ecosystem coherence (small weights) -- real but minor; never let formatting checks dominate a quality score.

Composite = weighted sum, times a multiplicative penalty for detected anti-patterns (e.g. 5% off per flag, floored at 50%) so a skill cannot check boxes its way past structural rot.

Writing Fixtures (the heart of the suite)

For each skill, build a fixture set of prompts with expected routing outcomes:

• Should-trigger: realistic first-person user phrasings, NEVER naming the skill. If fixtures contain the skill's name or its description's exact words, you are testing string matching, not routing.
• Should-NOT-trigger: the hard negatives -- prompts that belong to the skill's fenced neighbors. Routing quality is defined at the boundaries; "write me a poem" tells you nothing about a data-skill's precision, "explore this csv" might.
• Self-contained: every fixture must be fully answerable as written -- no references to attachments or context that the fixture does not contain.
• Distribution matters: mix terse and verbose, novice and expert phrasing. A fixture set drawn from one author's writing style overfits to that style.

Grader Design

• Anchored rubrics: for each dimension, define 5 score bands (0.0-1.0) with a concrete example of an artifact at each band. "0.3 looks like THIS" beats "score holistically". Unanchored judges drift run to run.
• Graders return structured output (score + one-line evidence), never prose to be re-parsed.
• Calibrate the grader itself: score a handful of artifacts you have hand-graded and check agreement before trusting it at scale.
• Position-bias check for head-to-head comparisons: evaluate both orders, flag disagreements.

Pass Bars and Badges

Set thresholds tied to consequences, not aesthetics:

• Gate (blocks release): composite below ~60-70, or any critical dimension (triggering) failing outright.
• Tiers/badges (communicates quality): e.g. >= 90 reference-grade, >= 80 production-ready, >= 70 functional-with-gaps. Tiers only mean something if some artifacts fail them.
• For catalog-wide ranking, pairwise comparison (Elo-style: each skill starts at the corpus median, wins/losses against gold-standard examples move the rating) is more stable than absolute scoring, because judges are better at "which is better" than "how good is this".

Regression Discipline

• Freeze the fixture set per release cycle; a fixture set edited while iterating on skills is a moving target that hides regressions.
• When a skill changes, rerun its own fixtures AND the hard negatives of its neighbors -- the most common regression is a description edit that starts stealing a sibling's traffic.
• Track composite scores over time per skill; investigate any drop, even if still above the bar.

Improvement Triage

When results come back with multiple failing dimensions, fix in weight-times-effort order: description rewrites (triggering) are cheap and heavily weighted -- do them first; restructuring for scope is medium; formatting fixes are cheap but nearly worthless -- do them last, if ever.

Fences

• Hand-evaluating one skill draft against a rubric right now: use skill-judge (this skill designs the machinery; that one is the manual instance of it).
• Evals for LLM-powered application features in production code: use llm-evaluation.

Usage

/evaluation-methodology $ARGUMENTS