We’ve all heard it. The more experiments your run, the more successful your website will be. But how do you know if simply running more experiments is the winning conversion rate optimisation (CRO) strategy for your business?
It’s not always simple to increase your experimentation velocity. Firstly, you need enough website traffic, and then you need to build your CRO team, meaning additional budget and finding people with the right skills to hire.
Jeff Bezos, CEO of Amazon.com, credits his company’s success to CRO; “Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day”. So it must be right, right?
The question stands. Will doubling your number of experiments, double your conversion increase? The answer is no. You need to understand which experiments are delivering quality, before increasing the quantity.
Optimising Your CRO Program
What type of experiments are driving the most value for your company? Should you focus on lots of smaller experiments, or just a hand-full of big experiments?
My experience tells me to do both – it’s the 80/20 rule. Spend 80 percent of your time on 20 percent of your experiments – these will be your high effort tests, and the remaining 20 percent of your time delivering 80 percent of your tests – these will be your low effort tests.
However, all websites are different, and so are their CRO programs. Optimising your program will identify the right mix of experiments that deliver the highest impact for your website. Like any good experiment, in order to optimise your CRO program, you firstly have to ensure you’re collecting the right data. Here are the key strategic elements you’ll need in place to analyse and optimise your CRO program.
A prioritisation framework not only allows you to score your experiment backlog, providing a methodology behind which experiments to run next, it also allows you to analyse the performance of your experiments once they’ve be run.
The most well known prioritisation framework is PIE (potential, importance and ease):
- Potential: (0=low, 5=high) likeliness of the experiment to win, user and/or revenue impact, and portion of your audience targeted
- Importance: (0=low, 5=high) alignment with business priorities, deadlines, or key stakeholder buy-in
- Ease: (0=high, 5=low) the effort required to run the experiment and implement the winning variation
This framework will give your experiments a score between 0 and 15, with a higher score representing a higher priority. This will not only allow you to easily identify your next best experiment, but will also later provide opportunity to analyse the performance of your CRO program.
Download: Free experimentation backlog template (including our simple prioritisation framework).
Storing Experiment Insights
It’s easy to finish an experiment, send out the results, and move on to the next. After all, speed is everything. But ensuring you have stored your experiment results in a methodical and searchable system is imperative to a scalable optimisation program. This will allow you to build your library of learnings and insights.
Some AB testing tools have built in workflow and knowledge base capabilities, such as Optimizely X, SiteSpect and VWO. There are also specialised CRO management and workflow tools, such as Effective Experiments. Alternatively, Jira, Trello, and Basecamp (just to name a few), have all been successfully used to manage large scale CRO programs. At the end of the day, a well thought out spreadsheet is a good start.
Analysing Your CRO Program
Now that all your experimentation results are readily available, you can sort and segment your experiments to see which have delivered the most impact for your business.
Ease of execution is only one attribute to analyse your program. You could also look at where the experiment was run – home page, landing page, cart checkout etc. Or, take a look at conversion triggers such as urgency, clarity, or distraction.
Now that you know which experiments are delivering the most value, you can now focus on scaling your CRO program. This is why it’s important to understand quality, before you should invest in quantity.