What it is for:An A/B experiment plan is a short, agreed document written before you compare two versions of a service with real users. You walk away with a signed-off, one-page plan: a single hypothesis, two clearly described variants, a main and a guard metric with their current values, an allocation agreed with an analyst, and a stop rule with the decision each result will trigger.
Who it helps:Service owners, designers and analysts with two credible versions of a page, message or step and enough traffic to compare them.
When to use it
You have two credible versions of a page, message or step and want to know which works better for real users.
The service has enough traffic or volume for a difference to show within a few weeks.
A decision is contested and the team wants evidence it agreed on in advance.
How to use it
Write one hypothesis: the change, the effect you expect and why.
Describe variants A and B side by side, changing one meaningful thing.
Choose one main metric and one guard metric that must not get worse.
Agree the split, sample and stop rule with an analyst.
Write what each result will mean, get sign-off, and freeze the plan before launch.
Tip:Write the plan in plain language that a manager outside the team could read in two minutes.
Credits and sources:Original Tiremesù worksheet and wording. Further reading: Microsoft Research https://www.microsoft.com/en-us/research/articles/patterns-of-trustworthy-experimentation-pre-experiment-stage/
Full guide and film:https://www.tiremesu.com/guides/a-b-experiment-plan
