Conducting experiments is not easy, but it can help your business to grow much faster! Unfortunately, hasty decisions and actions can not only delay the desired result but also harm your company. Thus, an unsuccessful experiment can disrupt the stability of the project, as well as cause discomfort to employees and spoil user experience. Check our guide, to avoid this kind of problems, and conduct experiments like a pro!
Step 1. Figure out a problem or gap
Before you start conducting an experiment you should ask yourself a simple, yet very important question: Why am I doing this? Experiments are good for you business, but they put a lot of pressure on development by complicating the code and business logic. So before you start spending any funds on the experiment you should know for sure why taking the risk.
We highly recommend figuring out a problem or gap area you are trying to improve first. This will not only help to keep idea generation focused, but will also increase the chances that the changes implemented will have any effect on your metrics and KPIs, which the golden calf of any business manager.
Example
We have an online store and 80% of customers add goods to the basket and never come back. Let’s think about how we can increase the incentive to finish the purchase and decrease the number of abandoned carts.
Step 2. Develop a hypothesis
Once you know the problem, it is time to brainstorm the solutions to solve it. Feel free to ask what all Team members think since different roles might look at the problem from the different angle, so great ideas might come from unexpected places. Besides, it increases the Team motivation greatly.
Once you have a list, think about a couple ideas that you like best. It is a good practice is to validate the ideas with the development Team to know how much effort it might take to implement a feature and whether there are any technical limitations. This will help you to prioritize the features right.
Example
Well, customers probably forget about their list or don’t have enough incentive to buy from us. There are many ways to try bringing customers back: you can send them emails & push notifications. Let’s focus on emails since we already have them all. What should we send them? Maybe a discount would work. And what about free delivery? Let’s send some of them a free delivery coupon and offer others 10% discount. And let’s have a group, who would receive a simple email-reminder, which doesn’t cost us anything and has no additional perks.
Step 3. Learn how to validate it
Any idea is good, but if you don’t know how to measure the efficiency, you will never be able to learn whether it was efficient or not. Remember in step 1 you figured out what problem you were trying to solve? It’s time to focus on that KPI and make sure that the results of the A/B test are measurable. Besides, you might want to put a benchmark to know what result you will consider as a positive.
Example
Currently, we have 80% of abandoned carts. We expect this figure to decrease after we introduce our emails. We would consider the result successful in case the % will drop to 70%.
Step 4. Document the requirements
The next extremely important step, which is skipped quite often, is documentation. One can’t but admit that having requirements documented eases things up for both business and development.
In cases with experiments, it is a must to document it. Having different behavior of the same feature at the same time complicates the business logic a lot and after a few experiments sometimes it is extremely hard to know what the original behavior was.
Make sure there is a professional person responsible for documenting changes and keeping track of all A/B tests making sure there are finished in time, requirements are written down and metrics are collected for them. Usually, Project Manager or Business Analyst is responsible for that.
Example
We want to test different emails. Project Manager is to create a ticket for each of the behavior types and write down what the experiment is, how long it is conducted and what aims we are trying to achieve.
Step 5. Do the A/B tests
Now it is time to conduct an A/B test, don’t forget to segment Users properly and have a control group with no changes. You will be comparing the results of the test groups against the control group. Mmm… smells like science!
It is important to set the limits to the size of the control groups and the time the experiment of the test. Usually, the experiments are conducted on not more than 30% of your audience. The ideal time for an experiment is 1 months, but not all businesses can’t afford to wait for so long, so in some cases, it lasts a week or two, but with large audiences taken in a game.
Example
Let’s put 70% of our Users as a control group – they will continue using the current product with no changes. 15% of the rest of the Users will be sent a free shipping coupon and another 15% will receive 10% coupon. In 1 month we will finish the A/B test and check the % of the abandoned carts in each of the groups.
Step 6. Analyze it right
Though it is very tempting, you should never analyze the results of the A/B tests before it finished. It is like having a wrapped gift on your desk, which you can’t open until Day X. Once you have the test ready, don’t jump with the conclusions: make sure that there are enough people in each segment for results to be credible. If there are not enough people there, maybe you should wait for more time.
Quite often there’s no a clear winner. In this case, you might choose one of the scenarios yourself or push the test candidates to the extreme, e.g. offer 20% discount instead of 10%. There are also cases when you decide to make no changes and test some other solutions instead. Though it gives you a bitter feeling, remember, that negative result is also a good result, since now you know for sure that it wouldn’t work for your business.
Example
Let’s assume that in the control group the % of the abandoned carts was 78%, in group A with the free shipping the result was 75% and in the group B with 10% discount it was 77%. The best decision here is to think about other incentives to decrease the amount of the abandoned carts, e.g. give them special discount right on the cart page. So you create a new A/B test and your experiment continues.
As you see, conducting experiments is a very creative journey. It is good for your business to test a different hypothesis. The only thing you need to keep in mind is that it always complicates the development and might lead to bugs and additional refactoring, so make sure to test only the best candidates.
