Mobile paywall A/B testing: sample size, setup, and decision rules
Plan a mobile paywall A/B test with verified sample-size examples, conversion windows, billing checks, and clear stopping rules. Includes a free calculator.
A useful mobile paywall A/B test starts with a question, a stable audience, and enough time to observe purchases. It does not start with a universal number of paywall views. This guide shows how to write a test plan, estimate the traffic you need, check the purchase path, and decide what the result supports.
Worked example: a 5% baseline conversion rate, a 20% relative improvement, 80% power, and a two-sided 5% significance threshold require approximately 8,158 unique users per variant under the fixed-horizon method below. At 1,000 new eligible users a day, a practical schedule is 21 collection days plus a seven-day conversion window: about 28 days in total. These are planning assumptions, not a prediction that the variant will improve conversion.
Write the decision before designing the screen
Suppose a reading app wants to test whether explaining offline reading before the purchase options helps new users subscribe. The control keeps the existing explanation; the treatment shows a concrete offline-reading example. Both use the same store products, prices, trial eligibility, entry trigger, and purchase implementation. This isolates an explanation change without turning the test into an unreviewed pricing migration.
A test may intentionally compare complete designs rather than individual elements. That is a valid decision if the question is which complete design to use. It will not tell you which changed element caused the difference. State that distinction before interpreting the result.
| Plan field | Illustrative decision |
|---|---|
| Population | New, non-subscribing users who reach the same onboarding checkpoint. |
| Assignment | Equal random allocation, with one stable assignment per user. |
| Primary outcome | First paid subscription within seven days of first eligible exposure. |
| Minimum useful effect | A move from 5% to 6% conversion: one percentage point, or 20% relative lift. |
| Guardrails | Purchase failures, restore failures, complaints, and access errors, with intervention rules agreed before launch. |
| Stopping rule | Reach the planned sample in both groups and allow every user's conversion window to mature. |
Use a trial-start outcome instead only if trial starts answer your decision. A seven-day paid-conversion window is unsuitable for a fourteen-day free trial. Define when the clock starts, which products count, and how you handle refunds and existing subscribers. Keep these definitions with the experiment revision so a teammate can reconstruct the result.
Calculate users, then calculate time
The paywall test calculator estimates two equal-sized groups using a normal approximation for independent binary outcomes. Its statistical method follows UBC's two-proportion sample-size calculator. It uses 95% confidence and offers 80% or 90% power. It is a fixed-horizon planning tool; it does not reproduce every experimentation platform's decision model.
For the reading-app example, 5% multiplied by 1.20 gives a 6% target. The estimate is 8,158 users in each group, or 16,316 total. Dividing by 1,000 new eligible users a day gives just over sixteen days of recruitment. The calculator rounds collection to complete weeks, with a two-week minimum, and then adds the conversion window. Here that produces 21 plus seven days. Weekly rounding is a scheduling choice, not a guarantee of a valid experiment.
Now change the assumption: with a 10% baseline, the same 20% relative lift targets 12%. At 90% power, the method estimates 5,142 users per group. The different result is why “a few thousand views” is not a defensible universal sample target. Both numeric examples were independently checked against UBC's calculator on October 6, 2026.
Daily traffic means new eligible people across both variants. A person opening the same paywall ten times does not supply ten independent observations. If only one quarter of your app's daily users reaches this paywall for the first time, use that eligible subset when estimating duration. Actual group totals may differ slightly under random allocation; confirm that each group reaches its target.
What to do when traffic is too low
At 100 eligible users a day, the first example takes far longer. A longer test may also span changes in acquisition, app versions, seasonal behavior, or the offer itself. More calendar time is not automatically more useful evidence.
Revisit the decision. A larger meaningful redesign can justify planning for a larger effect, but choosing an optimistic effect merely to shrink the sample does not make the result trustworthy. A more frequent outcome can answer an earlier funnel question, but it cannot silently replace paid conversion. You can also use interviews and usability sessions to find confusing language or broken interactions before committing scarce traffic to an experiment.
If the sample cannot distinguish the improvements you care about, report that limitation. “We did not detect a difference” and “the designs are equivalent” are different conclusions. Do not keep adding days after inspecting a disappointing result unless your analysis method explicitly supports that stopping behavior.
Verify assignment and billing before launch
Make a short device checklist. A new user must see the assigned design; reopening the app must preserve the intended assignment. An existing subscriber must receive the appropriate access. A canceled, pending, interrupted, or failed purchase must not become a successful conversion. Restoring an old purchase must restore access without counting as a new sale.
Record assignment, eligible exposure, product identifier, purchase outcome, and the authoritative purchase reference in the systems that own them. Avoid joining records solely by a display label such as “Annual”: labels can be localized or reused. Include the variant revision so that a published edit cannot silently look like an unchanged experiment.
Compare observed group sizes with the planned allocation. A substantial sample-ratio mismatch can indicate broken assignment, filtering, or missing telemetry. Investigate it before treating the apparent lift as a product effect; Microsoft's SRM guidance explains this diagnostic. An approximately equal split alone does not prove that all other tracking is correct.
Monitor safety without calling an early winner
Monitor purchase and access failures from the start. Stop or roll back a broken experience when users are harmed; statistical discipline does not require leaving a billing defect live. Record why you intervened and treat the interrupted run accordingly.
For a conventional fixed-horizon test, repeatedly checking significance and ending the test when it looks favorable can inflate false positives. A sequential or Bayesian platform needs its own documented decision rules. Do not mix its “chance to win” with the calculator's frequentist confidence threshold. Microsoft's during-experiment guidance discusses accounting for repeated measurements and early peeking.
Keep variants and eligibility stable. If you discover a misleading description or mismatched product, correct the customer problem, record the change, and decide whether a fresh run is needed. Quietly editing the treatment halfway through leaves an ambiguous result even if the analytics chart looks continuous.
Read the result against the original question
First reconcile eligible users, assignments, exposure, and mature outcomes. Then report each group's rate, the absolute difference in percentage points, relative lift, and uncertainty using the test's chosen analysis. Review guardrails before deciding on rollout. Segment exploration may reveal useful hypotheses, but searching many countries or acquisition channels for a positive subgroup creates another multiple-testing problem. Treat unplanned subgroup findings as follow-up questions.
A conversion win also does not prove a revenue or retention win. If the design changes which subscription people buy, follow the commercial outcomes that matter after the initial window. Preserve an inconclusive or negative result with the same care as a positive one: it tells the next person which question was tested and what the evidence could resolve.
Choosing tools for this workflow? Use the Superwall and Nuxie comparison, Adapty and Nuxie comparison, and RevenueCat and Nuxie comparison to work through vendor fit, billing ownership, and the checks to run before adoption.
Run the workflow in Nuxie
Nuxie's Experiment guide explains reviewing variants and a success definition, publishing, starting a test, and pinning a device for QA. The test's saved decision settings govern its results. The standalone calculator is an independent planning aid, not a statement that Nuxie's result engine uses this exact formula.
Use the measurement guide to distinguish unique exposed customers in an Experiment from presentation-based Experience totals. Start with one question, verify the actual purchase and access path, and keep the plan beside the result. That gives your next change a defensible starting point.