A/B Test Designer

/ab-test-setup

Design A/B tests and experiments that produce statistically valid, actionable results.

Usage

/ab-test-setup $ARGUMENTS

What I Need From You

1. What are you trying to improve, and what change are you considering?
2. Baseline conversion rate and traffic volume to the page or flow
3. The smallest improvement worth detecting
4. Timeline constraints and any past tests in this area

Core Principles

1. Start with a hypothesis -- a specific prediction grounded in data, not "let's see what happens".
2. Test one thing -- a single variable per test, or you will not know what worked.
3. Statistical rigor -- pre-determine the sample size; do not peek and stop early.
4. Measure what matters -- a primary metric tied to business value, secondary metrics for context, guardrails to prevent harm.

Hypothesis Framework

Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metric moves].

Weak: "Changing the button color might increase clicks." Strong: "Because users report difficulty finding the CTA (heatmaps and feedback), we believe making the button larger with contrasting color will increase CTA clicks by 15%+ for new visitors, measured as click-through from page view to signup start."

Test Types

• A/B -- control vs one variant; most common, easiest to analyze
• A/B/n -- several variants; needs more traffic
• Multivariate -- combinations of changes; needs much more traffic and careful analysis
• Split URL -- separate URLs; good for major redesigns

Sample Size and Duration

Inputs: baseline rate, minimum detectable effect (MDE), 95% significance, 80% power.

Duration = required sample x variants / daily traffic. Run at least 1-2 full business cycles (usually 1-2 weeks); avoid running so long that novelty effects and external factors pollute results. If traffic is low, test bolder changes with larger expected effects.

Metric Selection

• Primary -- the single metric that calls the test, directly tied to the hypothesis
• Secondary -- explain how the change worked (time to click, scroll depth, plan distribution)
• Guardrails -- things that must not get worse (revenue, retention, support tickets); stop if significantly negative

Example, pricing page test: primary = plan selection rate; secondary = time on page, plan mix; guardrails = support tickets, refund rate.

Designing Variants

Keep the control untouched during the test. Make the variant a single, meaningful change bold enough to detect, and true to the hypothesis. Dimensions to vary: message angle, value proposition, layout, visual hierarchy, CTA copy/prominence/placement, information order, social proof type. Document each variant with a screenshot or mockup and the reasoning for why it should win.

Traffic Allocation

50/50 standard; 90/10 or 80/20 for risky variants (slower to significance); ramping for technical risk. Ensure users see the same variant on return visits.

Running the Test

Pre-launch checklist: hypothesis documented, primary metric defined, sample size calculated, duration estimated, variants QA'd, tracking verified, stakeholders informed.

The peeking problem: checking results early and stopping at first significance produces false positives and inflated effects. Pre-commit to the sample size, or use a sequential testing method if you must monitor. Do not modify variants or add new traffic sources mid-test.

Analyzing Results

1. Did you reach the planned sample size? If not, results are preliminary.
2. Statistically significant? Check p-value and confidence intervals.
3. Practically significant? Compare effect size to your MDE and implementation cost.
4. Secondary metrics consistent with the primary?
5. Guardrails intact?
6. Segment differences (mobile vs desktop, new vs returning) -- but beware cherry-picking.

Output

A complete test plan: hypothesis, test type, duration and per-variant sample size, variant descriptions, primary/secondary/guardrail metrics, success criteria, and a documentation template so learnings feed a reusable test repository.