Stats Engine

What is the stats engine?

When GrowthBook queries your warehouse and gets raw per-variation metric values, it needs a statistical model to turn those numbers into a decision signal. This is the job of the stats engine. GrowthBook supports two engines — Bayesian (the default) and Frequentist — and you can choose per experiment or set an organization-wide default.

Bayesian engine (default)

The Bayesian engine treats the true effect as a probability distribution. It outputs:

• Chance to Win — the posterior probability that a variation is better than the baseline. Easy to explain: “there is an 89% chance this change is an improvement.”
• Expected Uplift — the expected relative change in the metric.
• Credible Interval — the Bayesian analog of a confidence interval.

The Bayesian engine is well-suited for teams that want to make incremental shipping decisions based on probability. You do not need a predetermined sample size — you can stop when the chance to win crosses your threshold (typically 95%).

Frequentist engine

The Frequentist engine uses classical hypothesis testing. It outputs:

• p-value — the probability of observing this result (or more extreme) if the null hypothesis is true. A p-value below your significance threshold (typically 0.05) means statistical significance.
• Confidence Interval — the range that would contain the true effect 95% of the time across repeated experiments.

Use the Frequentist engine if your organization already standardizes on p-values, your data science team requires it, or you are publishing results externally where p-values are expected.

CUPED variance reduction

CUPED (Controlled-experiment Using Pre-Experiment Data) is a variance reduction technique. If you have pre-experiment metric values for the same users, you can use them as a covariate to remove noise that is unrelated to the treatment.

The result: tighter confidence intervals and faster experiments — you reach statistical significance with fewer users, or equivalently, you can detect smaller effects at the same sample size.

GrowthBook implements CUPED automatically when you provide a pre-experiment metric in the experiment settings. The pre-experiment metric must be:

• The same metric (or a closely related one) measured before the experiment started.
• Available for most of the users in the experiment.

Sequential testing

Sequential testing (also called “always-valid inference”) lets you look at results at any time during an experiment without inflating the false positive rate. GrowthBook’s Frequentist engine supports sequential testing via the mSPRT (mixture Sequential Probability Ratio Test) method.

To enable sequential testing:

1. Open the experiment settings.
2. Under Stats Engine, select Frequentist.
3. Toggle Sequential Testing on.
4. Set your tuning parameter (default 5000 — leave as-is unless you have a specific reason to change it).

When sequential testing is on, GrowthBook adjusts the confidence interval to remain valid regardless of when you stop the experiment.

Choosing an engine

Consideration	Bayesian	Frequentist
Ease of communication to stakeholders	High — “89% chance to win”	Lower — requires explaining p-values
No required pre-planned sample size	Yes	No (fixed-horizon)
Safe early stopping	Yes (always-valid posterior)	Only with sequential testing on
CUPED variance reduction	Yes	Yes
Required by your data science / stats team	Sometimes	Often

🧪 Bayesian chance to win is easier to act on

For most product teams, Bayesian “chance to win” is the most intuitive decision signal — no need to explain p-values or significance thresholds. Switch to Frequentist if your organization standardizes on p-values, your statisticians require it, or you are publishing results externally. Both engines support CUPED.

What does the Bayesian "Chance to Win" metric represent?

The p-value of the experiment The posterior probability that the variation is better than the baseline The percentage uplift in the primary metric The width of the confidence interval

What does CUPED do in GrowthBook?

It splits traffic between variations It reduces metric variance using pre-experiment data, leading to tighter confidence intervals It runs the assignment query faster It validates that guardrail metrics are not regressing

When is sequential testing most useful?

When you want to analyze results only once at the end of the experiment When you need to look at results at any time without inflating the false positive rate When you want to use Bayesian inference instead of Frequentist When your experiment has more than two variations

Which stats engine should you choose if your organization requires p-values?

Bayesian Frequentist Either — both produce identical p-values CUPED

Submit

Mark as complete