Think you got a few clever messages to send your audience, but not sure which one to use? This is where A/B Testing can come in handy. A/B Testing (known as bucket testing or split testing) can help marketers test different messages and see what speaks to their audiences. OneSignal's A/B Testing tool might be just what you're looking for to try this out. We're trusted by over 650K+ developers to get their messages out and many of them A/B Test using our platform.

Why Is A/B Testing Important

So let’s say you have Notification A and Notification B. How will users react to each of your messages? Which message out of the two will not only result in the most engagement, but also bring in the most sales? A/B Testing helps answer all of these questions.

OneSignal's Templates feature within our testing tool makes it easy to experiment with whatever you need to.

When A/B Testing push notifications, Notification A will have its own title, image, and message. Notification B might have the same title and image as Notification A, but a different message. How will the difference in the message, even with the same title and image, influence click-through rates, engagement, and ultimately sales revenue?

That’s what we’ll find out through A/B testing and comparing the results.

What Should You Test With A/B Testing?

To A/B test your push notifications, your variants should be slightly different from one another. The message could be worded differently, for example. An emoji might be included in one variant, but not in the other. The following are a few examples of what can be A/B tested in your campaigns:

  • Slight variations in message wording
  • Emojis (either the presence of which, or the usage of different ones)
  • Images
  • Sounds
  • Action buttons

Since you cannot test different segments or delivery times, A/B testing on the above variables is best done on the same user segment at the same time to ensure minimal bias in your results.

How Do You A/B Test Push Notifications?

Running an A/B test is just like sending a "New Message" by specifying a set of users to send to, entering the message text, and selecting the delivery options. The only difference is that, when starting an A/B test, you use a Template for each A/B variant. You can either select an existing Template or create your own template, then modify the contents of each template based on the variable you want to test. If you want to test between two different images, you would use them in these templates while keeping the title and message the same.

After you schedule and run your test, you should get a report on how many clicks each notification received from your sample audience. OneSignal's reporting tool instantly shows how well one notification has performed compared to another.

During an A/B test, your sample audience will receive only one of the two notifications. Some users will receive Notification A, and others will receive Notification B, but not both. Once you see a notification performing better than the other, you can send that notification to the rest of your user base.

Check OneSignal's step-by-step instructions on how to conduct an A/B test for push notifications.

Pitfalls To Avoid While A/B Testing

Before you begin your A/B test, you need to know the common pitfalls that might render your A/B test useless. Some marketers make the following mistakes:

  • Testing more than one variable at a time
  • Not segmenting your users based on demographics and buying behaviors
  • Testing a too small of a sample size before calling the results
  • Not paying attention to overall analytics, including revenue

For example, a variant could be the clear winner in getting more clicks, but revenue actually plummets. This may be due to clickbait titles that are low in relevance to their real offer.

In other instances, A/B tests may fail because the test results were biased due to poor sampling, or the sample size was just too small and therefore the A/B test ended too soon.

Hopefully this helped you gain more knowledge and insight into the world of A/B testing and how it can help make your marketing strategies more effective! For more information on push notifications and how OneSignal can help you please visit us here!