April 24, 2022 - Reading time: 3 minutes
A/B testing is a method to find out which variant of a text or graphic converts better. When you want to change an aspect of the design, rather than going in blindly or going with your gut feeling, A/B testing lets you make changes that perform better, or at the very least as well as the existing design.
Your success with A/B testing depends on a few factors
A conversion can be defined as any action that is desirable. For example, a landing page can use sign-ups as a conversion metric.
The conversion rate is the percentage of participations that resulted in a conversion. It can be calculated as conversions / participations. Note that while we can calculate the conversion rate for different variants without any complex calculations, it won’t tell us anything about the confidence or the statistical significance of the result.
A/B tests are actually a case of multivariate testing with two cases, namely A and B. Depending on the method used, you can easily perform split tests across an arbitrary number of variants.
A participation happens the moment you roll the dice and decide on which variant to use. This can be when the user loads the page, or when you are sending emails.
Server-side participation is suitable for traditional web applications, such as those written in PHP, C#, Python or Java. When you receive a request from a user, you make the changes to the response you send, and then record the participation in your database. This allows you to serve the same user with the same variant, and prevents a single user from knowing about the variants.
For client-side participations, the client knows about all the variants, and can configure itself to any of them. After a variant is picked randomly, the client reports to the server (via API calls, analytics etc.) which variant it has chosen.
Client-side participation is suitable for static websites, SPAs, mobile applications and games.
There are different ways of performing A/B testing.
The Bayesian method provides estimates of the conversion rates as probabilities. This means that from the start of the split test until the end, you always have an idea about the estimated conversion rates for each variant. As you collect more data for the test, these estimates get more and more accurate.
Conversion rate estimation with Bayesian split testing is commonly done using the beta distribution. You can pull numbers from the beta distribution using random.betavariate(a, b) or numpy.random.beta(a, b).
// TODO: Write this section
Below are some examples of A/B testing in Python.
Here’s how you can calculate conversion rates in Python.
participations = 500
conversions = 210
rate = conversions / participations
# The conversion rate is 42%
print(f"The conversion rate is {rate * 100}%")
import numpy as np
# Variant A
# Participants and conversions
Ap, Ac = 500, 200
# Variant B
# Participants and conversions
Bp, Bc = 550, 230