Goals and results

Read Experiment results and choose what happens next.

Experiment results show whether a change achieved the improvement you set out to make. You can compare each variant with control, see whether other outcomes got worse, and decide which version new customers should receive.

Nuxie uses the test’s saved success definition to make that comparison. Changes to the Experience’s conversion goal don’t change the Experiment’s metric.

Success definition

Work with Nuxie to define success before publishing the Experiment. Tell it what should improve, what shouldn’t get worse, and how much evidence you want before choosing a winner. For example:

Nuxie includes these choices in the plan. Review them with Nuxie before publishing. After publication, the Experiment card shows the plan before you start the test. The values above are an example; use a duration and sample size that suit your app’s traffic and the change you’re testing.

Primary metric

The primary metric is the event you want to increase or decrease. Use conversion rate when you care how many customers take an action: it counts each exposed customer once. Use event count when repeated actions matter, such as how many workouts customers complete.

For example, a subscription event can measure the proportion of customers who subscribe, while a completed-workout event can measure repeated activity. Send your app event after the action succeeds. Internal Journey reports aren’t success metrics.

The Experience’s conversion goal is configured separately.

Guardrails

Guardrails measure other outcomes you don’t want the change to harm. Give Nuxie an event and the maximum relative change you’ll accept in the wrong direction. An offer might improve subscriptions, for example, while making fewer customers finish onboarding.

A 5% limit on a decrease is relative to the control result. If 40% of the control group finishes onboarding, a fall below 38% exceeds that limit.

A plan has one to ten guardrails. Each must use a different event from the primary metric and the other guardrails.

Measurement window

The plan sets how long the test should run and the minimum sample size for each variant. The duration can be up to 90 days. Reaching the end of the window alone isn’t enough: Nuxie also checks that each variant has received the required number of customers.

Decision thresholds

The minimum relative improvement says how large a change needs to be to count as a success. A 10% improvement on a 20% conversion rate means reaching 22%, not 30%.

The chance-to-win threshold sets how certain the comparison must be. It must be at least 95%. Ask Nuxie to include both thresholds in the plan.

These settings mean a small lead isn’t automatically a successful test. A variant also needs enough data, the required improvement, and acceptable guardrail results.

Results

Open the published Experience’s Experiments tab and find the test. Its card shows each variant’s metric, its difference from control, the uncertainty in the estimate, and its guardrail results.

The comparison counts unique customers who reached the Experiment, including holdouts. An assignment earlier in the Journey doesn’t count on its own, and repeated presentations don’t add more people. Variants and allocation explains when exposure is recorded.

Nuxie may still be collecting data, have too few customers to decide, or find that no variant meets your success criteria. It can also report that a guardrail has been breached. These results explain why the variant with the highest primary metric might not be a suitable winner.

Customers included in results

Device pins, fallback traffic, and customers seen in multiple variants are kept separate from the main comparison. Nuxie also checks whether the actual traffic split differs unexpectedly from the percentages you configured.

If those checks find a problem, Nuxie won’t report the comparison as a success even when one variant has a higher result. A selected variant’s loading or presentation failure still counts as an exposure; the presentation and error data show what happened afterward.

Change the analysis

Expand Traffic and analysis and click Edit analysis to change the primary metric event. Enter the new event and a reason for the change.

The original plan stays available alongside the amendment. Changes made after launch are marked post-hoc, meaning the metric was revised after the test began. This keeps the new analysis distinguishable from the comparison you originally planned.

If you want a fresh test with separate results, use Restart test or Copy test. Results from different runs aren’t combined.

Pick a winner

Click Pick winner on the Experiment card, choose a variant, and confirm. Nuxie ends the test and sends new arrivals to that variant. Customers already in a Journey keep the variant they started with.

You can choose a variant before the results are conclusive. That changes what customers receive, but it doesn’t establish that the variant improved the outcome. Read the comparison and guardrails before making that decision.

You can also finish the measurement or change traffic separately under Traffic and analysis.

End without a winner

End without winner records an inconclusive result and keeps the current traffic setting. Use End and restore to finish as inconclusive and send new arrivals back to the original control instead.

Change traffic

Pause traffic sends new assignments to control without concluding the test. Resume traffic returns them to your configured allocation.

You can also choose Restore original to use the original control, or Use a variant to send new assignments to one particular variant. These actions change delivery while keeping the test’s measurement state as it is.

Traffic changes apply to new assignments. Pick winner is the action that both ends the test and selects the variant for future arrivals.