Sample Ratio Mismatch Checker Online — Free Sample Ratio Mismatch Checker Tool
CodexFuse Team · 2026-11-05T10:26:19.455Z · 9 min read min read
Run a free chi-square check on your A/B test split. Catch sample ratio mismatch before a broken test ships a false winner.
A sample ratio mismatch checker is a free online tool that runs a chi-square test on the traffic split of your A/B test and tells you whether the visitor counts in each variant match the split you intended. If you planned a 50/50 test and the tool finds 5,214 visitors in variant A against 4,786 in variant B, it flags the imbalance as a sample ratio mismatch (SRM) — a sign your randomisation is broken and your results cannot be trusted.
Most teams never check. They run a test for two weeks, see a clear "winner", ship it — and never realise the split was skewed from day one. An SRM does not mean your variant is bad. It means the test was never fair.
What Is a Sample Ratio Mismatch?
A sample ratio mismatch happens when the actual number of visitors assigned to each variant of an A/B test differs significantly from the intended split. In a planned 50/50 test, a 52/48 split across 10,000 visitors is not random noise — it is evidence that something in the assignment, redirect, or tracking is systematically favouring one variant over the other.
Random assignment is the foundation of every A/B test. If it is biased, the two groups are no longer comparable, and any conversion difference could come from the audience rather than the change you tested. That is why SRM is treated as a test-killer: one significant mismatch invalidates the test, however convincing the numbers look.
Check for SRM within the first 24 hours of launching a test, not at the end. A broken split caught on day one costs you a day. The same bug caught after three weeks costs you the entire experiment.
Why SRM Silently Wrecks A/B Test Results
SRM is dangerous because it is invisible in the metrics most teams watch. Conversion rate, revenue per visitor, and significance can all look healthy while the split is broken.
- Biased samples: one variant receives more returning visitors or desktop traffic, so the groups differ in intent before the test starts.
- False winners: a variant can "win" by 10% or more purely because it received the better audience.
- Wasted traffic: every visitor sent through a broken test is a visitor whose data you must throw away once the SRM is discovered.
- Compounding errors: shipping a false winner bakes the mistake into your site, and later tests are measured against a polluted baseline.
How small a skew needs to be is the uncomfortable part. As your sample grows, ever smaller deviations become significant:
| Total visitors (50/50 test) | Smallest split that flags SRM |
|---|---|
| 1,000 | Around 53.5 / 46.5 |
| 10,000 | Around 51.0 / 49.0 |
| 100,000 | Around 50.4 / 49.6 |
How the Sample Ratio Mismatch Checker Works
The sample ratio mismatch checker on CodexFuse runs the same chi-square test a statistician would run, without any setup. It supports 2 to 4 variants and custom expected ratios, covering standard A/B tests as well as multi-variant and uneven-split designs.
- Enter visitors per variant: the actual counts your analytics recorded — for example, 5,214 for A and 4,786 for B.
- Set expected shares: enter the split you designed, such as 50/50, 70/30, or 25/25/25/25 across four variants.
- Run the check: the tool computes the chi-square statistic and p-value, then gives a plain verdict: SRM detected, or split looks healthy.
Behind the scenes, the tool compares each variant's observed count with its expected count, squares the difference, and divides by the expected count. The results are summed into one chi-square value and compared against the critical value for your number of variants. The chi-square calculator walks through the same calculation.
A p-value below 0.05 means the imbalance is unlikely to be chance — but it does not tell you why it happened. Treat the verdict as a fire alarm, not a diagnosis. The cause still has to be found in your assignment, redirect, or caching setup.
A Real SRM Catch: The Checkout Test That Nearly Shipped a Loser
Consider a fictional outdoor-gear store, Northwind Outdoors, testing a one-page checkout against its standard three-step checkout. After 12 days the test had 10,000 visitors and a result that looked like a triumph: the one-page checkout converted at 4.6% against 4.0% for the control — a 14% relative lift.
Before shipping it, the lead analyst ran the split through the sample ratio mismatch checker — and the visitor counts told a different story:
| Variant | Expected visitors | Observed visitors |
|---|---|---|
| A — three-step checkout (control) | 5,000 | 5,214 |
| B — one-page checkout | 5,000 | 4,786 |
The chi-square is (5,214 − 5,000)² ÷ 5,000 plus (4,786 − 5,000)² ÷ 5,000 = 9.16 + 9.16 = 18.32. With one degree of freedom, the critical value at 95% confidence is 3.84, so a chi-square of 18.32 (p below 0.001) means the split was decisively not random.
The cause was found within an hour: a CDN edge rule cached the control page for returning visitors, so they were never re-randomised and piled into variant A. Returning visitors also convert at roughly twice the rate of new visitors, which explained the entire "lift".
The team excluded the test pages from the cache, re-ran with fresh bucketing, and the split landed at 5,003 / 4,997 — a chi-square of 0.0036, no SRM. The honest result: the one-page checkout actually converted 1.8% worse than the control. Without the SRM check, the store would have shipped a losing checkout to every customer.
Once your split is clean, the next question is whether the result itself is real — the A/B test calculator checks whether your observed difference is significant or just noise.
When Should You Run a Sample Ratio Mismatch Checker?
Check at four moments, not just when something looks wrong. The earlier you check, the cheaper the fix.
- Within the first day: run the check once you have a few hundred visitors per variant. Assignment bugs show up early.
- After any deploy: a release, CDN rule change, or redirect edit mid-test can break assignment without touching your experiment code.
- Before calling a winner: make the SRM check a mandatory gate in your sign-off checklist, alongside the significance calculation.
- When segments look odd: check the split inside that segment too — an overall healthy split can hide a broken segment.
A broken A/B split wastes weeks of traffic and ships false winners. CodexFuse hosts your site fully managed on credits, so your experiments run on fast, stable infrastructure with no server babysitting — and the SRM checker is free to use whenever you need it.
Get Started with CodexFuseHow to Fix SRM Once You Find It
Finding SRM is the easy part. The cause is almost never the random number generator — it is usually a caching rule, a failed redirect, bot traffic, or double-counted page views. Fixing it means working through the assignment path in order, from the visitor's browser back to your server.
- Pause the test and stop making ship decisions on its data. The results are invalid until the split is fixed and the test re-run.
- Check assignment logs and test the redirect across browsers and devices, including mobile Safari and in-app browsers.
- Audit cache rules for the test URLs, exclude them from caching entirely, and filter bots from the analysis.
- Re-run with fresh bucketing and verify the split is healthy in the first 24 hours before letting the test run its course.
Never "correct" SRM by re-weighting the results after the fact. Adjusting the numbers to compensate for a biased split assumes you know exactly how the bias worked — you do not. Discard the test, fix the cause, and re-run.
What an SRM Checker Cannot Tell You
Honest limits matter here, because a clean SRM result can create a false sense of security.
- It detects imbalance, not the cause: the tool flags the symptom. Finding the broken redirect, cache rule, or bot flood is still your job.
- It does not validate conversion tracking: your split can be perfectly balanced while your conversion tracking misses half the purchases on one variant. Pair the SRM check with a tracking audit.
- It needs adequate sample sizes: chi-square results are unreliable when any variant's expected count is below 5, so very small tests will not give a trustworthy verdict.
Used within those limits, the checker is the cheapest insurance in experimentation: thirty seconds, no cost, and the difference between shipping a real improvement and a statistical artefact. The GA4 tracking checker can also verify that your analytics is recording the traffic your test depends on.
Frequently Asked Questions
What is sample ratio mismatch in A/B testing?
Sample ratio mismatch is a statistically significant difference between the intended traffic split of an A/B test and the actual visitor counts per variant — usually caused by broken randomisation, redirect, or caching.
How much imbalance counts as SRM?
It depends on sample size. At 1,000 visitors a split around 53.5/46.5 is significant; at 10,000 visitors around 51/49 is enough; at 100,000 visitors even 50.4/49.6 flags.
Can I still use results from a test with SRM?
No. A significant SRM invalidates the test because the groups are no longer comparable. Pause it, fix the cause, then re-run with fresh bucketing.
How many visitors do I need before checking for SRM?
A few hundred visitors per variant is a sensible minimum. Below that, the chi-square test can only detect an extreme imbalance. Check within the first day, and again before calling a winner.
Does the checker work for tests with more than two variants?
Yes. It supports 2 to 4 variants and custom expected ratios, covering multi-variant tests and uneven splits such as 70/30 or 25/25/25/25.
Is the sample ratio mismatch checker free?
Yes. Enter your visitor counts per variant, set your expected shares, and run the chi-square check — no installation and no subscription required.
Check the Split Before You Trust the Winner
Every A/B test rests on one assumption: that the two groups were genuinely comparable. A sample ratio mismatch breaks that assumption silently, and the cost is not a failed test — it is a false winner shipped to every customer. Run the free sample ratio mismatch checker in the first 24 hours of every test and again before you call a winner.
If the infrastructure behind your tests is part of the problem — caching quirks, redirect failures, servers you have to babysit — move your site to CodexFuse. Hosting, security, business email, and backups run fully managed on pay-as-you-go credits, with no subscriptions and no surprise bills. Check your current test split today.
