How do I estimate sample size and MDE for my test?

Select a primary metric and baseline rate or variance, choose a minimum detectable effect (relative or absolute), set alpha and desired power, then use a sample size calculator. For continuous metrics, you need an estimate of standard deviation; for rates, use baseline conversion. Revisit estimates if traffic mix changes materially.

How long should a web A/B test run?

Run until you reach the precomputed sample size and cover at least one full business cycle (e.g., a week). Stop early only if you planned a sequential method with clear boundaries. Avoid very long runs that span multiple seasonal shifts or heavy cookie churn, which can bias allocation and outcomes.

What if traffic is low—can I still experiment?

Yes. Increase effect size by testing larger changes, pool similar pages to share traffic, target higher-funnel metrics with more events, or run non-inferiority tests. Consider staggered rollouts or bandits for exploration, but still define success criteria and guardrails.

When should I use multivariate testing instead of A/B?

Use multivariate or factorial designs when you need to estimate interactions between elements and you have ample traffic. Otherwise, iterate with a sequence of A/B tests to keep the factor space small and maintain statistical power.

How do I pick guardrail metrics and set stop rules?

Choose guardrails that reflect user experience and system health, such as error rate, latency, bounce, and unsubscribes. Define acceptable ranges and pre-specify stop rules for data quality issues (e.g., SRM), severe UX regressions, or safety thresholds being breached.

How do Bayesian and frequentist approaches differ in decisions?

Frequentist testing controls long-run error via p-values and fixed alpha/power, typically with fixed-horizon or sequential designs. Bayesian testing reports posterior probabilities and credible intervals, enabling decisions at defined probability thresholds. Both require preplanning and clear decision criteria.

How do I detect and handle Sample Ratio Mismatch (SRM)?

Compare observed allocation to the planned split using a chi-square or similar test. If SRM is detected, pause the test and debug instrumentation, bot filters, redirects, and cookie/session handling. Do not trust results from an SRM-affected test.

A/B Testing Playbooks: Practical Templates, Examples, and Checklists

Jump to section

Overview

These A/B testing playbooks turn vague ideas into disciplined experiments. You’ll find repeatable templates for hypotheses, sample size, QA, instrumentation, and decision rules—so teams can move faster without cutting corners.

Each playbook is scenario-based: pricing, onboarding, emails, paywalls, navigation, and more. We cover frequentist and Bayesian decision-making, guardrail metrics, SRM checks, and sequencing tests to avoid interference. Use them as building blocks to scale an experimentation program that balances speed and rigor.

Who it’s for

Growth marketers owning website or app conversion.

Product managers prioritizing experiments at scale.

Analysts designing metrics, power, and stop rules.

Designers ensuring variants meet UX and brand goals.

What you will gain

✓

Ready-to-run test plans with hypotheses and KPIs.

✓

Sample size, MDE, and power calculators explained.

✓

Templates for QA, instrumentation, and launch docs.

✓

Playbooks for pricing, UX, onboarding, and email.

Key Takeaways

Actionable points curated for this category.

Start with a sharp hypothesis

State the lever, audience, and expected outcome. Pre-define the primary metric, directionality, and risks to isolate causality.

Design for power, not hope

Compute sample size from baseline, MDE, alpha, and power. For continuous metrics, use variance; for rates, use baseline conversion.

Ship with solid QA and instrumentation

Validate event firing, variant parity, performance, and accessibility. Exclude bots/internal traffic and confirm allocation accuracy.

Run disciplined tests

Commit to a fixed-horizon or sequential plan. Cover full business cycles, monitor guardrails, and avoid peeking-driven false positives.

Analyze beyond headline uplift

Check heterogeneity by segment, secondary effects, retention, and absolute impact. Investigate novelty and carryover before rolling out.

Document and operationalize

Use briefs, pre-mortems, launch criteria, and rollback plans. Maintain a searchable learnings library to inform future bets.

A/B testing playbooks

Who it’s for

Growth marketers owning website or app conversion.

Product managers prioritizing experiments at scale.

Analysts designing metrics, power, and stop rules.

Designers ensuring variants meet UX and brand goals.

What you will gain

Ready-to-run test plans with hypotheses and KPIs.

Sample size, MDE, and power calculators explained.

Templates for QA, instrumentation, and launch docs.

Playbooks for pricing, UX, onboarding, and email.

Featured Articles

Lifestyle Product Photos for Ads: A/B Test 10 Scene Variations Without a Reshoot

All Articles

Lifestyle Product Photos for Ads: A/B Test 10 Scene Variations Without a Reshoot

Key Takeaways

Start with a sharp hypothesis

Design for power, not hope

Ship with solid QA and instrumentation

Run disciplined tests

Analyze beyond headline uplift

Document and operationalize

FAQ

How do I estimate sample size and MDE for my test?

How long should a web A/B test run?

What if traffic is low—can I still experiment?

When should I use multivariate testing instead of A/B?

How do I pick guardrail metrics and set stop rules?

How do Bayesian and frequentist approaches differ in decisions?

How do I detect and handle Sample Ratio Mismatch (SRM)?

Create better visuals faster with Pixflux.AI

A/B testing playbooks

Who it’s for

Growth marketers owning website or app conversion.

Product managers prioritizing experiments at scale.

Analysts designing metrics, power, and stop rules.

Designers ensuring variants meet UX and brand goals.

What you will gain

Ready-to-run test plans with hypotheses and KPIs.

Sample size, MDE, and power calculators explained.

Templates for QA, instrumentation, and launch docs.

Playbooks for pricing, UX, onboarding, and email.

Featured Articles

Lifestyle Product Photos for Ads: A/B Test 10 Scene Variations Without a Reshoot

All Articles

Lifestyle Product Photos for Ads: A/B Test 10 Scene Variations Without a Reshoot

Key Takeaways

Start with a sharp hypothesis

Design for power, not hope

Ship with solid QA and instrumentation

Run disciplined tests

Analyze beyond headline uplift

Document and operationalize

FAQ

How do I estimate sample size and MDE for my test?

How long should a web A/B test run?

What if traffic is low—can I still experiment?

When should I use multivariate testing instead of A/B?

How do I pick guardrail metrics and set stop rules?

How do Bayesian and frequentist approaches differ in decisions?

How do I detect and handle Sample Ratio Mismatch (SRM)?

Related Categories

Create better visuals faster with Pixflux.AI