A/B testing has been one of the frequently used terms since the beginning of this series. The reason is that it’s the real eye-opener to the change that will improve your website conversion rate. Everything we have done right from the beginning has been preparation for this moment, and with the last one, we were just a step behind. Now, it’s high time to test all the hypotheses formed in the previous article.
But before you start splitting your audience and testing some hypotheses, you need to know
- What is Bayesian Statistics, and why it’s essential?
- How long does it take to determine the credibility of a hypothesis?
- What type of test for your hypothesis – Split test or Multivariate test?
- What mistakes to avoid while running hypothesis tests?
Let’s get to it.
What is Bayesian Statistics?
Principles and standards usually guide experiments and procedures. Without these underlying principles, it will be hard to validate the credibility of the outcome of a test. The same applies to A/B hypothesis testing. Bayesian statistics is used in split hypothesis testing to eliminate guesswork and ensure accuracy.
Let’s say in a split test of a baseline to treatment, and you realize that the treatment has a 20% conversion rate when the baseline has a 10% conversion rate. The baseline is the original page, and the treatment is the new page with the change proposed in the hypothesis.
It’s just a few hours into the test, and only 50 visitors have been sent to each page. Literarily, anyone would suggest that the treatment is 10% better than the baseline. However, it’s difficult to conclude with that little traffic. You can’t really say the total 100 visitors you sent to both pages is a good representation of your audience.
For this reason, you need Bayesian Statistics to determine the probability that the treatment will beat the baseline. Bayesian statistics is a math principle that helps to determine if, infact, the leading variation is better than the competition statistically.
The Bayesian statistic is not the only framework for determining which is better. There are others, but the commonest among them is the Frequentist Statistic. However, Bayesian is most preferred because it’s straightforward, very intuitive, supports quick decision-making, has a high confidence level, and uses less complicated terms like p-value and significance value. The incredible thing about this framework is that you can peek at your data while testing is in progress.
The mathematics behind Bayesian Statistics include Bayes theorem, Beta and Gamma distribution, Posterior, and other complex equations you wouldn’t want to learn. For this reason, I suggest you use a Split testing tool that uses the Bayesian Approach. Examples of such tools include VWO, AB tasty, and Convertize.
Time Taken to Validate a Test Result
How to know the accuracy of a conversion rate test and when to stop a conversion rate test are the two most frequently asked questions anytime this subject is being taught. The first one is answered in the first part. Let’s deal with the second here.
The duration of conversion rate testing depends on four factors which are;
- The average number of daily visitors
- Number of variants
- Your current conversion rate for control
- The difference between your new expected rate and the current conversion rate, also known as the lift
There are calculators you could plug in these values to obtain your testing duration. These tools include the VWO’s SmartStats, HubSpot’s A/B testing kit, and the Unbounce Sample Size & Test Duration Calculator. Some of these tools also provide the sample size required. It is the total number of visitors needed to run a test in conversion rate optimizations.
The mathematical relationship between the two is:
Where T = Test duration, S = Sample Size, and N = Average number of daily visitors
However, no matter the number of calculated days for a test, do not run a test for less than a week. Calculators can output duration of fewer than seven days for a test based on the parameters input.
It has been found that tests that lasted less than a week can’t be trusted to give an accurate result because some businesses have special sale days. Besides, shopping behavior during weekends is quite different from those shown during weekdays. For this reason, ensure your testing lasts for at least one or two weeks.
Types of Hypotheses Tests in Conversion rate Optimization
There are two major types of tests used in conversion rate optimization. They are
-
Split or A/B testing
Split testing occurs when you compare two versions of a web page. It’s always one treatment against the original baseline page. The difference between the two variants could be a change in one element or changes in multiple components.
For example, you could decide to test a treatment whose signup button has a green color with the control that has a blue signup button. Also, it could be testing multiple changes against the original.
The website traffic is divided into two equal parts, and each is sent to the two pages to discover which one has a higher conversion rate.
-
Multivariate testing
When you make multiple changes in A/B testing, it’s challenging to tell which element caused the increase in conversion rate. For that reason, a multivariate test is usually conducted to know which part contributed to the growth by making many versions of your pages.
You could make each variant have one component change so that you can easily trace the reason for the increase in conversion rate. Your website traffic is shared equally between all the variants. Multivariate testing is not commonly used because it requires massive traffic and takes longer to ascertain its validity.
Mistakes to Avoid while Hypotheses Tests
Split and multivariate testing are simple when you use the right tools and follow all timing and sample size instructions. However, it can also become confusing and frustrating when you try some things during your tests. The following are mistakes to avoid while running an A/B or multivariate test.
- Performing multiple tests at once with the same traffic
- Not giving the test enough time to experience the complete business cycle, which usually leads to unreliable results
- Changing test parameters while the test is ongoing
- Getting discouraged and ending a test abruptly because the result is not going as proposed
- Running split testing on a new campaign. Wait, at least a week for the campaign to attract its traffic to measure the baseline conversion rate
Good Hypothesis Test Practices
- Learning from wrong hypotheses to build strong ones in the future
- Never stop testing. One test should lead to another
- For A/B testing, do not test for more than one component change at once
Conclusion
Conducting hypothesis tests is quite straightforward when all the questions posed in the introduction are answered. Make sure you know what test type is suitable for your hypothesis, calculate the sample size and duration required to get the most accurate result, and do not forget to use a testing platform that runs on Bayesian statistics.
We hope by now you know how to run a test. Run some split tests for your website and share your experience with us in the comment section. Also, feel free to ask questions if you need any help with hypothesis testing.
