OneSignal's Vice President of Product, Zack Hendlin, goes over Outcomes use cases, setting up outcomes on your site, and viewing/understanding analytics, attribution windows, data exports, and more. The transcript of the webinar can be found below.
Great. Let's dive into it.
Hi, my name is Zach. I'm the VP of Product at OneSignal. Today, I'm going to be walking you through some of our new advanced analytics capabilities. In particular, we're going to talk about outcomes and how outcomes are a powerful way of tracking different events that are important to you within your app and being able to understand if those come from push notifications, if those happen close in time to someone getting the push notification, or if they're sort of independent of that. And that's going to allow you to make better decisions as you refine your messaging as you build campaigns, as you engage your user base.
Can you all see my screen? If you can't see my screen, please post into the Q and A and we can fix them.
Okay, great. Then I'll continue.
So, we're going to talk through a couple of things today. First, how outcomes help you drive results that matter. Secondly, we're going to talk about some built-in outcomes. These are outcomes that don't require you to write any code. It just works out of the box. These are things like session and session duration. They tell you how long folks are spending in your app or on your site.
Then we're going to go into more detail about custom outcomes. These are really powerful because they let you add a snippet of code anywhere in your app or website that you want to track an important outcome. That could be someone adding an item to cart, that could be someone purchasing, that could be someone sharing a piece of content, signing up for an account, and we'll go through a couple of examples.
And then we're going to talk about how you actually set them up on web or on your website. We'll do a live demo there so you can see how it works end to end. And then we're going to talk about how you'd set that up on mobile. We'll go through a couple of nuances after that and open it up to Q and A.
What are Outcomes?
Great. So, what are outcomes? Outcomes let you measure the actions you care about across both your mobile app and your website. They let you figure out if those actions were coming from someone clicking the notification or were influenced. Influenced is the term we use to refer to an action that happens, not from someone directly clicking a notification but rather within some time period that you could define, between 15 minutes and 24 hours. So, we'll go in more detail on that.
But basically, outcomes are important things that you want to track along with your customer's journey or along your user's journey, and you'd like to be able to know and analyze them for every notification you sent to see if those notifications are driving an outcome. Now, you can also use this just as a kind of general purpose analytics tool for your app or website, separate from notifications as well.
Notifications are really easy to visualize and export. So, in every notification you send, if you're on a paid plan, you'll actually be able to see a breakdown of all the notifications that happened as a result of that notification. You'll be able to specify different time windows and all that sort of stuff.
They're available on all our major platforms, so Android, iOS, Web, as well as all of our commonly supported wrappers, so React, Xamarin, Flutter, and so on.
Use cases for outcomes are tracking and improving the performance of push notification campaigns. Another use case is App Analytics. So, how many people are signing up? How many people are adding a friend? How many people are coming back to my app on a regular basis? How many folks are needing to reach out to my support team within my app? So, you can instrument any of these outcomes that you are interested in tracking pretty easily. We'll talk about how to do that a little later.
There's also product analytics, right? So, let's say I'm an e-commerce app. I may want to track a couple of points along the user journey. I want to track when someone maybe browses into a category, let's say men's fashion, women's fashion, kid's fashion. And that gives me a sense of how they're going through my site.
And then maybe I want to know when they add an item to cart. I might even want to know the dollar value of items that they're adding to cart. So, we'll talk about outcomes with values. So, an outcome doesn't just need to be a single item, a single string. It can also have a value or a weight associated with it.
Outcomes and Advanced Analytics Use Cases
I might even want to know how many unique users are adding items to their cart. I can also do that with outcomes and we have a special type of outcome called Unique Outcome. And that will de-duplicate at a user or player ID level. So, let me see that kind of de-duplicated one.
And then finally, it's actually really helpful to use outcomes to quantify the value of your notification campaigns or your marketing efforts because you'll be able to say, "Hey, I sent this campaign. I sent this notification. I sent this automated messaging campaign. Here’s all the purchases, here's all the revenue associated with these campaigns that I did. Here's all the players of my game that I got to buy tokens as a result of some of these re-engagement campaigns or these push campaigns or this offer that I extended."
Great. So, a couple of examples of how other folks are already using advanced analytics and outcomes on OneSignal. So, a fitness app, Rumble, rewards people for exercising and it gives them coins that can be redeemed at a local business. Think of a coffee shop or a taco shop or something like that. Rumble wants to understand how users are spending their coins, how they're logging their workouts, and they want to use that to improve their product and fulfill a product analytics function so they could figure out what categories people are redeeming their coins on really easily.
Another example is a meal planning app where you put in the meals you're eating, how much water you're drinking, and all that stuff. They use outcomes to track when people are updating their profiles, when they're recording exercise sessions, and they're really able to get a much better understanding of how users are going through their app.
There was a question that Renee just asked about if we will make a recording of this webinar available after because I think she's home-schooling.
Yes, we will make a recording of this available after on the OneSignal YouTube channel within a couple of days of hosting it today. Thanks for asking the question, Renee.
Another example is a social game developer. Basically, they're trying to get their free users to upgrade to paid with additional lives or additional functionality. So, they want to track free to paid conversion so they make an outcome for upgrading to paid. They also track actually when a credit card or payment attempts fail. So, they track both successful transactions and failed transactions. And that can be useful actually because it means maybe your checkout flow is a little too complicated if a lot of transactions are failing or someone's abandoning. Or, it means maybe you're PayWay, payment gateway went down and you didn't even know that, but you're able to track that by saying, "Okay, hold on. We're seeing a spike in payment failures. We should dig into that and investigate."
Renee asked where she can get the link.
If you just go to YouTube and type in OneSignal, we at OneSignal have a page where you'll be able to access all of our previous videos. So, we have a channel on YouTube, and that's probably the best way to gain access to the recording of this.
And then finally, a grocery delivery app in Europe uses outcomes and they track Add to Cart. You can track both the event of adding an item to the cart as well as the dollar value of that item. They also track the eventual purchase of when folks actually pay and check out.
You could imagine you might even want to track more complicated things. So, if you're a grocery delivery app, you might want to track items by category. You may want to track how long people are taking in your app before they do checkout to figure out if you want to reduce the friction of checking out, as an example. So, that gives you a sense of how outcomes could be used and how they're already being used by a good number of apps out there.
I'll talk through a little bit more detail now of what we include by default as a set of outcomes and then the ones that you would use custom outcomes for. And then I'll go do a live demo on a website.
So, building outcomes cover three things. They cover clicks, how many people click on the notification. It's pretty straight forward. We've always offered that even before we shipped outcomes.
And then there's two other things that we built in: sessions and session duration. A session could happen independent of someone clicking on your notification because we have this idea of 'influenced outcomes' as well as 'unattributed outcomes'.
So, let's say I send you a push notification but you don't click on it. However, you open my app, let's say ten minutes after getting that push notification. We can see in the OneSignal dashboard all the sessions that happen within that time window, and we can do attribution. OneSignal will automatically do the attribution for me to say, "Was that someone clicking on the notification that caused that session? Or was it someone coming to me within close time proximity to getting a push notification?" That's what we call it 'influenced'.
You can view the breakout between Direct, Influenced and Unattributed. Unattributed are folks, let's say coming to your site or adding an item to cart or spending time on your site or app, but where that session is not created from them clicking on a notification, nor is it created in a time window up to 24 hours from receiving a notification.
So, clicks, session, and session duration. Session duration is just how long someone spends on your website or your app in that session.
One other thing that I'll point out here is you can actually determine the window that you want to use for influenced outcomes. So, for instance, if you are a news app or a media company and you send, let's say ten notifications a day with different news that's happening in the world, you actually don't want a 24-hour attribution window because you're really interested in if you're driving more immediate action and more immediate visits to your app or website. So, in that case, 15 minutes or one hour are probably better attribution windows because they more closely resemble; folks coming to your app or website, very close to when they received a message.
But if you're an eCommerce site, it's pretty unlikely that you're sending your users ten notifications a day. Maybe you send them one notification in a week, as an example. In which case, you'd actually be quite interested in a 24-hour attribution window because that's much more aligned to the frequency with which you're sending notifications.
Maria asked if Advanced Analytics is an extra product or if it's included in the plan.
So, Advanced Analytics is included in the starter plan and the pro plan. It's not included in the free plan. The reason for that is we have to store additional data, right? And that means that we need to have additional capacity to be able to support that. So, outcomes and advanced analytics are available on all of our paid plans, but they're not available on our free plan.
So, we recently shipped this on web. Actually, April 13th, we shipped this on web. It's been available on mobile for a number of months. Use of outcomes is a paid feature, but it's included already in all of our paid plans.
So now, let's jump over to a live demo where I'll show you on an example page that I have how I can send an outcome. So, what I would do is I would figure out, let's say it was an Add to Cart button or a purchase button or a sign up for a demo with our sales team button. On that button, I would actually say... I can actually show you here.
So, we have the Send Outcome button. And all we're doing, as we're saying, well, when someone clicks on that button, that button on click, I'm going to call this 'function'. You actually don't even need to do this function. You could directly call this one line of code. This is actually the one line of code that you need, which is OneSignal.sendOutcome. Here we're sending an outcome without a value because we were not passing number in there, and I'm going to have that fire when someone clicks on the Sen Outcome button.
So, as an example, I'm gonna send myself a notification so you can see how Direct Notifications work. So, in your Messages, send a new push and say, "Thanks for joining the webinar." So, you know it's all live all. Let’s say put in the time. It's 11:16 AM PST. If I wanted, I could make it pretty by adding an image. Let's see. Find an image from the web. Webinar. Great. Okay. So, this looks like a generic tech image. We'll grab that one. Cool.
I'm going to actually send it, in this case, only to my test devices because I don't want to send it to all my users. So, I'm going to create a segment. So, tests users. I have 18 users. You see it automatically updates. Great. I'm going to send that notification.
Before I send that notification, I'm actually going to close this screen, this tab that has my website open. I'm going to do that because I want this to be a direct outcome to show you what direct outcomes are. So, if my site is already open, then that session already exists. So, the outcome is not attributable to this notification that I've gotten. It's instead already existed. So, I'm closing it so that you can see what it's like as a newly created session. Hit Send Message. I just got a notification on a different screen. I clicked that notification. Now, I've created this session that is directly attributable to me having clicked on that notification.
So, I'm going to hit Send Outcome and I'm going to hit it three times. And then I'm going to say, "sendOutcome with value 10." I'm going to hit five times. Great. So, I've just sent regular outcomes three times and outcomes with values five times.
Now, what I'm going to do is go to my delivery page where I'm going to be able to look at those results. So, I just sent this notification and I'm going to click into it. You'll see I get all the regular stuff, number of deliveries, number of clicks, all that stuff. That one click is me because I clicked it. I can split by Direct Opens, Influenced Opens. I could look at a longer time range. I could cut it by different platforms.
I'm going to add the web_click_send_outcome. You see right here, I told you I clicked it three times. The outcome name I used was web_click_send_outcome. I see three clicks that are directly attributable to me having sent that notification.
Now, I mentioned that I also sent this outcome with a value of 10 and I did that five times. So, you might be asking, "Well, great, I see this outcome with a value of 10. It shows five direct fires here. But how do I actually be able to add those values?" Let's say it's purchased amount or things added to the cart. So, what I'm going to do here is change outcome value from count to sum. And now you'll see OneSignal is automatically calculating for me the sum of those values.
Now, I can have these both appear at the same time. So, I can say 'outcome_value_10'. Here I can see that I had five of this fire and I had a sum of them, of their values that was 50.
Now, you might be asking, "Well, great. That explains Direct. What's Influenced?" So, let's now open that website again. I'm going to send three more outcomes. Here we'll say 'with value'. But you'll notice when I opened up my site this time, I just opened it up directly in my browser. I didn't create that section from clicking on a notification. It did happen though in close time proximity to clicking a notification. So, that's going to end up showing up as influenced.
So, I'm going to send three outcomes with the value and, let's say one outcome without a value. Great. We're going to close the session.
I'm going to give it a sec for this data to update. And now, you'll see those three outcomes with the value that I fired are showing up here. You see three of them and the sum of their values is 30. Great. I had one outcome without a value, here shown as web_click_send_outcome. As you see correctly showing up here, those are recorded as influenced.
If I wanted, I could export this data, all of my outcomes or selected outcomes, and it will email me the table that I see here. I also can go and change the attribution window in my settings if I wanted to between 15 minutes, 1 hour and 24 hours.
Now, one other question you might ask is, well, that's great. That tells me the outcome results from a single notification I send. Cool. I want a view of my whole app. I want to know how many add to carts are happening directly from all my notifications so I don't want to have to go into the analytics for each notification.
So, what you can do is you just hit the Delivery tab, and then you hit Outcomes as a sub-tab. This will show you the analytics across all notifications you've sent across your whole app for Direct and for Influenced and Unattributed, even if you didn't send a notification. So, this is aggregating across all of the outcomes that your app is firing, even if they're not attributable to a specific notification. So, this gives you the broader multipurpose analytics capabilities that go beyond just a single push notification.
So, we've shown you a Direct, Influenced, and you might be wondering about Unattributed. Unattributed is any time that someone comes to your site, which would be a session, any amount of time. Let's see. Session duration is where we record the amount of time. So, some of session duration, that's the number of seconds that folks spent on your side or in your app.
So, here we can see in this example, since this is a demo, much of the time that we have spent on the page wasn't necessarily from people clicking on push notifications or in close time proximity to getting a push notification. Great. So, let's go back to see how we might do this on other platforms.
So, just to remind you on web, if we want to send an outcome without a value, we just would omit this 18.76. And if we want to send an outcome with the value, we add whatever value we want after the comma.
On mobile, it's quite similar but slightly, slightly different where we actually have three different function names. So, there's OneSignal.sendOutcomeWithValue. That's me sending an outcome with a value. I specify similarly here. This is Java and Objective-C for Android and iOS respectively.
If I wanted to send a regular outcome, I would just omit the 'with value'. I would just say OneSignal.sendOutcome("name of the outcome". If I wanted to send a unique outcome, I would just say OneSignal.sendUniqueOutcome. That again will de-dupe at a player or a user level.
So, a little bit about attribution. When someone clicks on a notification and creates a new session, that means your website's not already open or your apps not already open, that is considered a direct session. Any actions that happen within that session are considered direct action. Only one notification is the direct source of a session.
So, as an example, if the app is closed, I send you a notification. You click that Notification A and open the app. When you're in the app, you get a second notification - B. Notification B will not be the direct source of that session. Any outcomes that happened in that session would be shown as direct from Notification A since Notification A was the notification that instantiated that session.
We talked about Influenced Outcomes and Influenced Opens. That's just when a user comes within a time window you specify, between 15 minutes and 24 hours, and you specify under settings on the OneSignal dashboard. And then unattributed is all the actions that happen that are not direct and are not within the influenced time window that you specify.
These custom outcomes allow you to instrument any event in your app that you want. We talked about the three types: regular outcomes, outcomes with values, and unique outcomes.
Outcomes with values are really useful when you have things like purchase amounts or let's say a user can invite their friends to your app. You might want to know two things. You might want to know how many people are getting invited, right? So, if one person invites five of their friends on one screen, you might want to say sendOutcomeWithValue. Your outcome name might be add_friend or invite_friend, and you might have a value of five because that person invited five friends.
You might want to use sendUniqueOutcome. If you want to know something like, how many of my users are inviting their friends? Even if I invite five friends today and five friends tomorrow, I only want to record that once. Then I would use sendUniqueOutcome, where even if that fires from the client, from your app or your website 5 times, 10 times, 100 times, we're only going to count the first time.
All outcomes that have values have two ways you can look at their metrics. You can look at how frequently they occurred, how many times they occurred. You can also look at the sum of those values. Of course, if you wanted averages or something, you would just be dividing the sum by the count.
A couple of notes: Clicks are reported for all SDKs. But session, session duration, and custom outcomes require you to be on the latest version of the SDK.
Now, if you have a website, you don't have to worry about that because your website will be pulling from the latest version automatically. It automatically updates as we release new versions of SDK. If you've done something super custom, then you may need to update it on your own. But for 99.9% of folks using us on their websites, it will just automatically update. If you're using this on your app, you need to update to the latest version of the SDK to be able to get this functionality.
Want to give a little bit of a sense of what's coming next. So, we'll be building out API support that will allow you to get the outcomes associated with a particular push notification. It hasn't shipped yet, but it's something that we're starting to work on.
Over time, we'll also let you get outcomes at a user level.
And coming quite soon, actually, in the next month or so is the ability to fire outcomes from in-app messages. You'll be able to do that without having to write any code. The way that will work is you might say, "Hey, I'd like you to rate me from 1 to 10. How likely are you to recommend this app to a friend?" And you might have 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. So, you might have rate_1, rate_2, rate_3, and so on. You'd have 10 little buttons. If someone taps 1, that would fire the rate_1 outcome. If they fire rate_10, we fire that outcome. From that, you actually could get something like net promoter scores to be able to understand how folks are rating your app and what your net promoter score would look like.
In terms of how you can learn more and get the details that you need, there's documentation available at https://documentation.onesigna..., and you can just search for outcomes or you can go to this URL. With that, happy to dive into any questions that folks may have.
I think Argae just posted a question: Is there a limit to the number of outcomes that an account or app can have? Is that number based on the level of your paid plan?
So, you can have up to 100 outcomes that are live. If you get past 100 outcomes, what we do is we'll automatically pause collection on the new ones. You can go say, "Hey, I don't want to track these old ones anymore," in your settings, just by going to the settings page and you can remove outcomes from being tracked. So, let's say you had some old metric that you don't care about anymore. You could remove that from your list of outcomes.
Now, very few folks should end up needing more than 100 outcomes. If you do, then that's something that we can open up additional outcomes for you on our pro or pro plus plans. But if your base plan is free, you don't have access to outcomes. If you are on a paid plan, you can fire up to 100 different outcome types or outcome names. If you're in need of more than that, you can get in touch with our support or sales team. We can raise limits there on some of our more enterprise-focused plans.
The second question was: There was a very clear use case for outcomes that are based on clicks. Are we able to make an outcome out of situations where they land on a particular URL, for example?
Third question was: Can these outcomes be used to create custom audiences of subscribed users such as folks who clicked on sendOutcomes that could be followed up with a more targeted notification?
So today, unfortunately not. That's a direction that we're moving in. But it obviously requires a good amount of engineering. That's going to take us a little bit of time. What you will be able to do in the quite near future is, when someone clicks an in-app message and fires an outcome, you'll have the option of adding a tag on that person. When you add a tag on that person, you can then retarget them very easily. You'll be able to do that without any code.
However, to answer your question about how you could retarget these folks, basically, what you would want to do are two things: you'd fire send tags and that will add information to that player's information, to that player's record, that user's record that you then could use to retarget them.
To answer Argae's question about how can I retarget people based on outcomes, short answers: You can't retarget them right now explicitly based on outcomes, but you could fire send tags, which is a method available in both our web SDK and our mobile SDKs. That will add a tag to that user. And then when you're sending notifications, either via our dashboard or via our API, you basically would say, "I want to send this message to users where item in cart is true or where page is visited, the tag page and it's for visited or something, is technology, something like that. Thanks for the great questions, Argae.
I'll jump into some of the other questions that folks asked.
Someone asked: Is there a way to collate clicked outcomes and number of messages for a given time to get a rough click-through rate?
So, we show a number of clicks both by notification you send, Monica, and we show a number of sends. So, if you just wanted to get a rough click-through rate over time, you could just export from the dashboard a list of the notifications you'd sent recently. Now, that dashboard view doesn't cover the underlying outcomes, but what you could do to correlate that is you could go to the Delivery page and then Outcomes. So, you go to the Delivery page and then Outcomes. Here, you would just select the same time window, and you'd be able to get all these outcomes that occurred over that time period. So, that might be a way that you could correlate them in a way that you were looking to.
Kevin asked about Amplitude and integrating these analytics.
So, we have an integration section on our documentation where we discuss some of the OneSignal events that you can send to Amplitude on your mobile device. These don't natively pass through to Amplitude right now. But if that's something that's interesting to you, that's good to know. Thank you for sharing that. It's a thing where we're always looking at what integrations we want to continue to invest in overtime. But I would direct you, Kevin, to this page where it walks you through how you would integrate your OneSignal events like notifications received, notifications open with sending those on to Amplitude.
Yeng asked: What are the best practices to integrate OneSignal into HubSpot.
So, it depends exactly how you're using HubSpot. They've got a marketing hub. They have a CMS offering. What we generally suggest folks to do is set an external ID in OneSignal to match up with your HubSpot user ID for that person. That will allow you to use things like our List Upload function to reach a set of people, target them based on all the data that you have on them in HubSpot. So, short answer there is, set the external ID in OneSignal, and there's a method for how to do that.
So, we have a blog post on it and it's covered here in our documentation. It's basically just this set external ID method in our SDKs. Here is how do you do it. So, basically, if they're subscribed, you call onesignal.setuserid and then you pass the user ID that you want to set. And then when those things match with your information on them HubSpot, we have this other thing, which is a list upload tool, where you can just upload a set of your HubSpot user IDs to send a message to them. You also can upend tags and other stuff. So, you can then sub-filter that data to target folks more specifically.
Stelios asked what the best way is to increase a click-through rate.
So, I'd encourage you to go to one single.com/blog, and there actually we have a lot of tips about how to increase click-through rates. In fact, we actually had an analysis of 118 billion notifications that have been sent and what some of the metrics were for what works best and what doesn't. So, as contacts, we send now north of 6 billion notifications a day.
If you go to this blog post about insights from 118 billion push notifications sent through OneSignal, we actually show you the percentage improvement of using different functionalities. So, if you use segments to reach users more specifically, we see a roughly 50% gain in click-through rate. So, if it was 1% click-through rate, that'd be 1.5% for our customers who have used segments.
We also walked through using data tags. We walked through AB testing. The really cool thing here is we actually have metrics about how impactful each of these steps are to take. So, that's what I would look at, Stelios, in terms of improving click-through rate.
And then finally, I guess the last question is... Two questions: one from Chris: How can we send this via the console?
So now, I just sent an outcome that was ask_webinar_question. Just to show you it really works, I'm going to go here to my Delivery tab, go to Outcomes. I'm going to update this. I'm going to refresh the page just because it can take us a sec. here you see ask_webinar_question. Now, it's shown as influenced because I already had that session open. I opened it on my own. I didn't open it from clicking a notification. So, that's why it shows up here as influenced. great question, Chris.
And then just to illustrate one more thing. If I wanted to send an outcome with a value, I would just say value is, let's say 9.99. I'll do two of those and then I can go back over my dashboard, ask_webinar_question, where I want the sum and I see the sum of those two values, 9.99 times 2 showing up here. Thanks for the question, Chris.
And then last question is here from Anatosh: Can you tell us about the integration using GTM for React Native mobile app?
So, if you just update your React, if you just pull down the latest version of React Native wrapper, all of this stuff will work in React Native as well.
And then you also asked Anatosh, do I recommend that you use user tags?
Yeah. So, as I was talking about before in the statistical insights from 118 billion push notifications, we actually talked about data tags here for apps or websites who have placed data tags on users. You need to be thoughtful about what the right data tags are, right? So, just adding a tag that has no meaning isn't going to get you a better click-through rate, but adding, "Hey, these people have added something to their cart. Hey, these people have read about finance news on my website instead of technology news." So, I'm going to send more specific targeted notifications about finance and stuff technology. When we see folks do that, we see on average a 16% higher relative click-through rate. That is if your click-through rate was 10%, it would now be 11.6%, as an example.
So, I want to thank everybody for taking the time to tune into those webinars. We'll make it available on our YouTube channel and it should be available in the next couple of days on our YouTube channel.
Thanks, everybody, for joining, and thanks for asking such great questions. I hope you have lots of success with OneSignal.
If you ever have questions, feel free, just to go to onesignal.com and you can click on Support, and we've got a great support team staff 24/7 to answer additional questions you might have.