A/B Testing for Publishers - A Practical Guide

In this A/B testing complete guide episode, we’ve compiled everything from what A/B testing is, its benefits, and its challenges. A/B testing, also known as split testing, has long been established in digital marketing to optimize web pages and elements therein. While A/B test has a goal of increasing Revenue Per Mille (RPM), the complicated process can challenge publishers. That’s why this video covers all the tips and tricks that can suggest ways for sites to get the jump on their competitors.

Key Takeaways

• A/B testing is the process of testing two or more variations of the same page to determine which elements are more effective in driving conversions — users are split equally between version A and version B, and after the test period the winning variation is rolled out to 100% of the audience.
• Publishers can use A/B testing in two key ways: testing ad layouts to find which yield higher click-through rates, and testing content to understand what their audience finds most valuable — both lead to meaningful improvements in page RPM over time.
• The best time to A/B test is during the month or quarter when marketing spend is lowest for the year — typically January for many publishers. This minimises revenue lost during the testing period when results are still inconclusive.
• A major challenge for publishers specifically is that most A/B testing software is built for digital marketers, not ad operations — meaning ad clicks inside iframes can't be tracked, and many ad networks explicitly prohibit the use of analytics software to measure ad clicks directly.
• Keep it simple: a news site with three ad units, six placement options, three size options, and six colour schemes creates a combinatorial testing nightmare. Narrow your variables, be clear on your goals, and run focused experiments rather than trying to test everything at once.

What is A/B testing, and how does it work?

Naomi: How often do you A/B test? All the time? Almost never? It seems a common mistake among smaller publishers is an over-reliance on intuition and personal preference — leaving A/B testing to the bigger players. Well, that's absolutely not the case, and we're going to tell you why. In this video we're going to go through all things A/B testing and how you can use it to improve your site. Let's get into it.

Naomi: A/B testing is the process of testing two or more variations of the same page in order to determine which elements of the page are more effective in driving conversions. Take a landing page for example — it usually operates during a limited, predefined period where users are allocated in equal numbers to version A and version B. Once this period of time is over, a winner is declared and 100% of the audience is then sent to the winning variation.

Definition — A/B testing: a method of comparing two or more versions of a webpage or ad placement against each other to determine which one performs better on a defined metric — such as click-through rate, page RPM, or conversion rate.

What can publishers do with A/B testing?

Naomi: There are really two key ways a publisher like yourself can use A/B testing to benefit your site. Firstly, you can embrace your inner digital marketer and A/B test ad layouts that yield higher click-through rates. Or you can use it to test which content is deemed newsworthy by your audience. These smaller, consistent experiments really add up — leading to improvements in all the important stuff, like page RPM. If you need a refresher on page RPM, do check out our video here.

Definition — Page RPM (Revenue Per Mille): the estimated revenue a publisher earns per 1,000 page views, across all ad formats on the page. A key metric for measuring the overall monetisation performance of a site.

When is the best time to A/B test?

Naomi: A common question asked is: when should I A/B test — is there an ideal time? And absolutely yes, there is. It's often said that January is the ideal time, as digital spend is usually lower in the wake of the festive rush. However, this is industry-specific, so for others January might be a busier month than usual. Regardless, the key takeaway is to try and conduct A/B testing during the month or quarter where marketing spend is lowest for the year. This is because it's inevitable that at some point during the testing process your website is going to lose ad revenue while you analyse the data — so don't worry if things take a dip. However, this isn't to say that you shouldn't do your best to minimise loss. Just remember: while your page RPM might decline, A/B testing when executed correctly can statistically increase your ad revenue substantially over the long term.

What are the main challenges publishers face with A/B testing?

Naomi: Of course, nothing is perfect. And while it's essential to A/B test to improve user experience and page RPM, there are a few challenges that can get in the way — however, many of these can be overcome.

Naomi: One such issue surrounds the software commonly used for A/B tests, which is usually designed for digital marketers. This can make it difficult for website owners to use. In particular, publishers will often face an inability to track ad clicks. Sites working with ad networks could face this problem for two primary reasons. Firstly, most ad networks include the creatives in the form of an iframe that doesn't support tracking ad clicks. And secondly, many of these networks have a programme policy that prohibits the use of analytics or software to measure ad clicks directly.

Definition — iframe: an HTML element that embeds one webpage inside another. Ad networks commonly serve creatives inside iframes, which creates a sandboxed environment that blocks external tracking scripts from measuring user interactions like clicks.

Naomi: Another issue is the lack of support for automatically creating variations — bit of a mouthful, I know. Let's think about a news website for a second. Typically they'd want to show around three ad units on a page. There are usually six or seven options for the placement of these ads, about two to three size options, and a further five or six for the ad's colour scheme. It's pretty clear how testing across all of these variables would be a nightmare. Unfortunately there are currently very few options that allow users to automatically estimate this kind of scale, making it arduous and time-consuming. Smaller websites, however, could cut down the number of test options for each page to save themselves time and money.

Summary

Naomi: Just remember — as valuable as A/B testing is, you don't want to over-complicate things for no reason. So be clear with your goals and get experimenting. And that's it — I hope this makes A/B testing a little less daunting. Check out our blog post below if you'd like to learn more, or otherwise you can book a chat with us via the link in the description box. If you enjoy these explainer-type videos, be sure to give us a like and subscribe — it really helps the channel reach more publishers. All the best, and get A/B testing!

This is an edited transcript of AdTeach, produced by Publift. The words are Naomi's own — lightly edited for readability (filler words, false starts, typos, punctuation). No claims have been rewritten or generated by AI.

‍