• All you need to know about Multivariate Testing

    Multivariate testing is a great way of determining what the best option for your website is when trying to increase conversion rates. It is especially useful when looking to figure out how different combinations of your web page’s elements work together. But if you haven’t been introduced to A/B testing, perhaps you should start by […]

    Multivariate testing is a great way of determining what the best option for your website is when trying to increase conversion rates. It is especially useful when looking to figure out how different combinations of your web page’s elements work together.

    But if you haven’t been introduced to A/B testing, perhaps you should start by reading our article on A/B testing before diving into this matter.

    Like any other digital marketing testing tool, multivariate testing needs to be used at the right time for the right purpose, otherwise, it might yield insignificant or non-essential results.

    And since there is more than one way to run multivariate testing, let’s find out which fits your strategy the best.

    Multivariate testing, definition and use

    Multivariate testing is often also called MVT. It is a testing method in digital marketing, which allows the user to combine various elements in different combinations and observe how the website’s traffic reacts to them.

    A/B testing vs Multivariate testing

    A/B testing versus Multivariate testing
    A/B testing versus Multivariate testing, Image courtesy of https://www.marketingprofs.com/

    What is different here from A/B Testing, is that we are testing multiple elements at once, and multiple variants. (In A/B testing, most often, you test 1 element in 2 variants or two separate pages altogether.)

    Multivariate testing isn’t a more advanced A/B test, despite what it might seem to non-marketing professionals.

    A/B testing example

    For example, while performing an A/B test, the subject of your test might be whether to put an image or not. And regardless of how many images you test, this wouldn’t turn into multivariate testing, but instead in A/B/n testing.

    The reason for that is, you are testing only a single element and not separate combinations of elements.

    A good example of A/B testing is this microcopy addition, which raised the conversion rate drastically.

    A/B test
    It’s free, microcopy A/B testing, Image courtesy of wordstream.com

    Multivariate testing example

    You can have as many variations as you want, but typically they vary between 3 and 8. These can look like:

    • Variation 1 – Banner 1, Button 1, Heading_1, Colour 1
    • Variation 2 – Banner 2, Button 2, Heading_1, Colour 2
    • Variation 3 – Banner 3, Button 3, Heading_1, Colour 3
    • Variation 4 – Banner 4, Button 4, Heading_2, Colour 4
    • Variation 5 – Banner 5, Button 5 Heading_2, Colour 5
    • Variation 6 – Banner 6, Button 6, Heading_2, Colour 6

    Like you can clearly see in the examples, some elements can stay the same, and others can change. In the case, shown above in the bullet points, we are testing only 2 headings, while the other elements are being changed with each variation. Another way to do things is to design a few elements and swap them around. That would look like this:

    1. Banner 1, Button 1, Heading 1
    2. Banner 2, Button 2, Heading 2
    3. Banner 1, Button 2, Heading 1
    4. Banner 1, Button 1, Heading 2
    5. Banner 2, Button 2, Heading 1
    6. Banner 2, Button 1, Heading 2
    7. Banner 1, Button 2, Heading 2
    8. Banner 2, Button 1, Heading 1

    This way, you get to design at least six different variations of your webpage, using only 2 elements. In the end, the one that gets the highest conversion rate wins.

    Multivariate testing pros and cons

    Not only does multivariate testing save a ton of time, but it also allows you to find new combinations by experimenting. However, like any other marketing method, multivariate testing offers both pros and cons.

    Pros

    • It can save time in some scenarios. (Imagine testing every single microelement on your webpage with an A/B test and running it for a month.) Multivariate testing does that at once. On the other hand, it takes more time to accumulate enough statistical significance. (Read more below in the cons.)
    • Measures interactions between all the parts of your web page that you include in the test. You can actually build a spreadsheet and break down the results, seeing what works best in which combinations. It is an insight you won’t get anywhere else.

    Cons

    • Multivariate testing requires a lot of time to actually reach the stage of statistical significance since you are testing a lot of different pages. Thus, it requires either time or a big traffic source. (If your webpage has a ton of traffic, you would reach enough visits in no time.)
    • For the time you can run a multivariate test (around 3 months), you could run 3 A/B tests. It is up to you to decide which might have a greater impact on your website’s optimisation process.
    • MVT is expensive. The reason for that is, you are actually playing around with the productivity and conversion rates of your website. Let’s imagine 5 out of the 6 variations perform very badly, and the 6th is doing great. If we average things down, you would eventually be losing revenue, due to the 5 poorly performing variations. Moreover, the more variations you are monitoring, the more resources you would have to use.

    When to use Multivariate testing instead of A/B or A/B/n testing?

    A general rule of thumb when performing website tests is to use A/B testing for large changes.

    For example, are you changing a major headline, banner, or your main Call to action button, which brings in most of your conversions? Then use A/B testing or A/B/n testing.

    On the other hand, if you are looking to change many small elements on the webpage, go for Multivariate testing.

    How to perform a Multivariate test?

    Multivariate testing with six combinations and three elements in sections, Image courtesy of https://support.google.com/

    You can read the entire Google Optimize instructions here. We will summarize in brief how to do it below.

    1. Go to google Optimize and log into your account.
    2. Go to the “Experiments” page and click on the “+” button to start. Select “Multivariate test”
    3. Think of a name for your experiment and type it.
    4. “Editor page” means the control variant of the page you are about to experiment with.
    5. Click the”Create” button in the right upper corner.

    Select a google analytics view that suits your needs. If you don’t have one or don’t know how to link them, you can read Google’s official guide here.

    How the process looks like, Multivariate Optimize testing
    How the process looks like, Image courtesy of https://support.google.com/

    From here on, you can start devising your brand new test variants. Go to the “Add a new Variant”. The window should look like this:

    Add a variant to an MVT
    Adding a new variant to an MVT, Image courtesy of https://support.google.com/

    Enter an appropriate variant that you would understand. A good guide for naming variants in Multivariate testing would be [element1#,element2#,element3#].

    For example, if you are testing headings, colours, banners, then your variants can look like this:

    • Heading 1, Green, Football banner
    • Heading 2, Blue, Football banner
    • Heading 2, Yellow, Basketball banner,

    and so on.

    Moreover, after you’ve saved the variants you can always edit them. The Google Optimize box looks like this.

    The Sections tab with the Editor button highlighted.
    Edit Headline Element of a section, Image courtesy of https://support.google.com/

    Visual editor

    The next step is to use the visual editor in Google Optimize. It is formed from two components, an editor panel, which you can see in the image above, and the app bar.

    You can read all about using the visual editor in Google’s instructions here.

    There are other tools aside from Google Optimize

    Despite our example is focused on Google’s testing tool, there are many other viable software solutions that you could use to perform successful Multivariate testing.

    Other Types of Multivariate testing

    multivariate-testing
    Testing different positions and colour elements in Multivariate testing, Image courtesy of https://abtastychina.cn/

    Now that you know what multivariate testing is, and how to start doing it, you should familiarize yourself with the options to choose from. Let’s start with the most basic form.

    #1 Full Factorial Multivariate testing

    The full factorial multivariate testing method or FF MVT is the most commonly picked and used among marketers. The main idea behind this test is to distribute the traffic that comes to your website equally between all variants that you are testing at the same time.

    Thus, if you are testing 5 variations, 20 per cent of your website’s traffic should go to each one.

    This is the recommended method for beginners and websites with decent traffic sources. For low traffic websites, see #3.

    #2 Taguchi Multivariate Testing (Taguchi Testing)

    Taguchi testing is a very old school method, which was first used as a preliminary test to determine whether other variants of a product should be manufactured. It is based on using a planning matrix, which consists of 8 steps.

    The main goal of Taguchi testing is to keep the difference (variance) between separate testing options as low as possible.

    It is a method that was initially developed for real-life scenarios and not digital campaigns or websites. Thus be advised to stray away from platforms suggesting that you use this method. Nevertheless, it isn’t harmful to know how it works.

    #3 Fractional Factorial Multivariate Testing

    Fractional factorial or also commonly called Partial factorial testing sends traffic only to a small group of variants. The remaining variants remain unexposed to traffic and the software calculates how they would have eventually performed if they had been exposed to the traffic, based on the performance of the exposed variants.

    This method allows for more variants to actually be tested, but on the other hand, doesn’t yield as accurate results as one would want them to for obvious reasons.

    Very advanced mathematical methods are used for the calculations of the potential conversion rates, but as you might deduce, empirical evidence always beats hypothesised statistics, regardless of how powerful the AI model is.

    This method is highly recommended for websites that lack the traffic to use Full factorial multivariate testing.

    Important Tips and Guidelines for a successful Multivariate test

    Multivariate testing multiple elements example
    Multivariate testing multiple elements example, Image courtesy of https://www.punchbuggy.com.au/

    Sometimes before starting the test it might be a good idea to try and determine whether some of the element combinations are redundant or simply won’t work. Since you have the opportunity to test a hundred variants, doesn’t mean that you should do it.

    • Limit yourself to testing the best options. (Unless you are performing Fractional Factorial testing.)
    • Always have a hypothesis. (This would limit you to testing a few options, ideally between five and eight variants.)
    • Learn to ignore inconclusive results. Sometimes we run a test for three months and get a difference of 0.01 % between variations. If this is the case, don’t waste any more resources on changing your website for such a small difference. Instead, ponder on what else could impact your conversion rates.
    • You can check out VWO’s multivariate testing duration calculator to try and determine how long you should be performing the test.
    • If you are running a website with lower traffic, but don’t want to use Fractional Factorial testing, go for Full Factorial testing, but limit the number of variants.

    Another very common mistake is to test more elements than needed. Perhaps the transparent background of the webpage wouldn’t have such a large impact as the heading, the CTA or the image. Prioritize your elements.

    Especially in the case of having less traffic than needed to reach statistical significance within a certain timeframe, you must learn to prioritize the right elements and weigh down the change of which ones could potentially lead to the highest ROI.

    Eliminate underperforming variants ASAP. By saying as soon as possible, we don’t mean that you should eliminate half of your variants on the second day. Wait for a minimum of statistical significance to be reached, and if a certain variant is greatly underperforming, then eliminate them and distribute the traffic evenly among other competitors.

    This would speed up the process, and you won’t be wasting time on losing variations.

    MVT favours design over copywriting. When performing multivariate testing, you should be well aware of the fact that it favours design over copywriting. And while both of these form an equally important part of your conversion rates and engagement, one should never be improved at the expense of the other, unless your results show that this is optimal.

    Thus, utilising multivariate testing, as well as alternating this approach with A/B testing your copy should yield the best overall results.

    Conclusion

    Multivariate testing is a fairly flexible method of determining how the elements in your website work together. Before you start, make sure to have a good idea of which the important elements are.

    Avoid testing unnecessary sections of your website, as depending on your traffic, it might take more than three months to reach statistically significant results.