Crazy Egg A/B Tests: Guide to Multi-Armed Bandit Testing

Now that you know exactly how your actual website visitors are experiencing your site, it's time to make data-backed changes that will help you increase your conversion rate. With the Crazy Egg A/B Testing tool, you'll be able to improve your user experience to find out what factors appeal to your audience. By creating variations of your web page(s), you'll be able to test which elements are the most crucial to motivating website visitors to convert.

Our tool lets you make changes to any number of landing pages on your website including Pricing, Demo, Signup, and Product pages. You can experiment with elements like your calls-to-action, headlines, copy, product descriptions, form fields, order of navigation links, button colors, and images. Change up wording, font size, colors, and word count with just a few clicks of the button to see how visitors respond compared to your original design. No code necessary! 

Crazy Egg A/B Testing Methodology

All A/B tests involve pitting two or more versions of a page against each other. With traditional AB testing, you would need to know the current conversion rate of your control and to decide how much of a change you want to detect in your A/B test. At Crazy Egg, we use a multi-armed bandit approach to A/B testing. Our algorithm constantly adjusts traffic to minimize any losses so you can quickly see positive improvements in page performance. 

The multi-armed bandit approach has four advantages over traditional A/B tests:

  1. You don't have to be a data scientist to conduct tests. You do not need to know your current conversion rate or pick a sample size before running your experiment.
  2. You can maximize returns. In a traditional A/B test, if you have one variant that is underperforming and losing you sales, you still have to keep going until the experiment concludes or you invalidate the data. However, because multi-armed bandit testing constantly adjusts traffic in response to performance, underperforming variants will receive less and less traffic without being shut off completely. This ensures that if the underperforming variant makes a comeback and turns out to be the most successful version in the end, your conversion rate is maintained consistently.
  3. Increased flexibility. Multi-armed bandit tests are more flexible because you can add new variants and stop underperforming variants while the test is running without invalidating the experiment.
  4. They are adaptive. Keep your ROI high by launching and maintaining ongoing A/B tests. Multi-armed bandit tests never need to be shut off, and are ideal for responding to changing customer tastes and minimizing false positives. 

What is the Multi-Armed Bandit Approach

The goal of the multi-armed bandit algorithm is to strike a balance between exploring different UX/UI approaches and exploiting winning setups.

At first, your traffic will be split evenly between Version A and Version B. As the experiment runs, the bandit algorithm will continuously analyze how each variant is performing. Using this information, our Crazy Egg system will change the fraction of traffic that each variation gets. High-performing variants will get more traffic and underperforming variants will get less traffic. As the experiment continues to run, the traffic distribution will continue to change until a winner is found. This adaptive approach to A/B testing ensures that your website won't suffer any losses while you experiment with how to increase your conversions. 


Jump to:

Plan your A/B tests

Still need help? Contact Us Contact Us