Outcomes and Mobile SDKs

OneSignal's Tech Lead and Software Engineer, Josh Kasten and Michael DiCioccio, covers OneSignal's approach towards Android and iOS SDKs as well as how our newest feature "Outcomes" came to be. The transcript of the webinar can be found below. Register for our webinars today!

[00:00]

Josh: So, welcome to our OneSignal webinar. This is mainly going to cover Outcomes. We're mainly going to focus on mobile - Android and iOS SDKs - how they work and the existing logic we've had with our SDKs and the changes that we're going to be making.

Just a quick overview, in case you're new to OneSignal, we're a push notification service. We focus on mobile as well as web push. Again, this is going to be mostly covering mobile, but eventually, we'll have the outcome feature in the future for our web push SDK as well. We also support email. The outcome is something that's really important to know, basically what users do in your app into response to notifications. So this is like a really good integration of analytics around what a user does afterwards.

I'll cover also our mobile SDKs or platforms like I was explaining for our session and our focus logic. I guess just to introduce myself a little bit more since it's the first webinar I've done, I'm Josh Kasten, our SDK tech lead. I work with Mike here as well.

Michael: Hi, I'm Michael. I'm a Software Engineer here. I started about six or seven months ago, and I work on the mobile SDK very heavily with Josh.

Josh: Cool. Yeah. I guess I didn't say before too. I've been here actually since the beginning of OneSignal about five years ago. So, I started all the SDKs. We work with George when he was more focused on engineering at the time and help to get everything up and going. Cool.

Michael: Yeah, let's get into it.

Outcomes Wrapper SDKs

Josh: Yeah. So, let's dive into it here. So, I explained about OneSignal already. So, I guess I pretty much covered. The slides are ready as well. Basically, for our mobile SDKs, we have an Android native SDK and iOS Native SDK. I indicated the versions here at which we added Outcomes. So, in iOS, that's 2.12.0 and for Android, that is 3.12.0. But as far as that goes, we'll have our wrapper SDKs as we call them. Around this, React need 3.5.0, Cordova 2.7.0, Flutter 2.2.0, Unity 2.10.0, Xamarin 3.6.0, and Corona 1.15.0. So, as soon as we release these versions, you will have the Outcomes features for both Android and iOS, which we plan on releasing sometime in the next week or two depending on how things go with our testing.

Michael: Right now, Androids released fully for all platforms. We're just waiting on iOS for the wrappers.

Josh: Yup. Correct. So, you could start playing around with this right now with our current latest SDKs before these versions, if you just want to test out Android and see how it's working today.

Sessions

So, before we dive into what changed about Outcomes and what happens through Outcomes, I want to explain what a session is and how our focus logic works because before Outcomes, we've had already a number of sessions of how long you spend in the app independent from the result of an application, just like general, so you can see users come back to my app very often. He or she has a large number of sessions and spends a lot of time in the app. So, that explains what a session is.

So, sessions basically start when the user opens the app, and the control is when we call a focus event to our backend to indicate that the session has started. And then, if a user does happen to background the app or leave it or turn off their phone or gets a phone call or something, if they leave the app for more than 30 seconds, we count that as the session being ended because they've probably been interrupted and they're spending time now doing something else in their on their phone. So, when they come back, it might be for another reason at that point.

Michael: And once they come back to the app, it would be a new session since the previous one ended.

Josh: Yeah. Just to cover the states here. So, when the app gets background, the user exits the app, it's not a focus. When they come back to focus the app, then we check to see if it's actually been 30 seconds. So, we end the session. And then we resume any focus time to keep extending that and know how long they've been spending in the app.

[04:54]

So, to cover that now, it’s basically just time tracked inside the app. We have an on focus endpoint that we send to the back end at this point. It’s only just for efficiency. So, we're not sending lots of network calls. So, every time the user goes in and out of the app, we're not tracking one second or five seconds. We wait until it's a cumulative of 60 seconds. That should be some significant amount of time being tracked.

And then when the app goes in the background, we stopped the unfocused tracking and we could potentially start a new session when the user comes back. So, another step between your-- it's greater than basically summarizing that.

Michael: So right now, that's the current workflow with our SDK when it's integrated into your application, where the session and focus logic is very independent of each other.

Outcome Events

Now, we'll get into Outcome Events a little and then we'll tie into both of the things we just talked about and explain how they're integrated together.

What is an outcome event? An outcome event is a way of tracking the intentions of a user after they've clicked on the notification where they've had notifications that they've received and had been within your application. It's not like an attribution window or a time window.

Triggering an outcome can be happening in three different ways. The first way being a simple notification click. So, you receive a notification, you click it. The other one being just opening the app in general. When you open up the app, you could have had notifications within a certain timeframe that influenced you in opening up the app. And then the other scenario was just the public methods that we created for sending different Outcomes. These being like a normal outcome, a unique outcome or an outcome with a value.

And then the other important thing to talk about with Outcomes is our session types. We talked about what a session was, but now we have to figure out how to determine what the session type is. This is how you came into the app or what notifications you have in the shade when you opened up the app. An example is just a direct being triggered by clicking a notification. And then an indirect session, you open up the app, goes in the foreground and you just have notifications in the background. You didn't click a notification to get into the app, but you've received them in a time window. Unattributed would be just opening up the app and having to notifications at all.

The next thing is showing just a very complicated flow chart of how we organized and implemented our logic for Outcomes. This is just to give you some insight on how technical it can get. We're not going to go into all of this, but as you see in the diagram, I've put a red rectangle around on focus delay, which is one of our events. If we look at this individually, this adds even more complexity into the logic for Outcomes with our session and focus tracking. But we're going to go forward and talk about more specifically the session types and how they're triggered.

So, you've opened up the app in the foreground by simply clicking a notification. This is a direct session. Very simple. And then at the bottom, you'll see a screenshot of how this would appear on the dashboard, where I've highlighted the direct column and you're looking at one click and one session duration.

Now, the next thing is our indirect. Our indirect session type occurs in a few scenarios. One of them being you have notifications that you've received in the last time window. This time window can be adjusted from the dashboard, your settings under analytics. You can change this time window to 24 hours, 1 hour, figure what the other options are, 30 minutes. There's a few other options. But right now, the default is 24 hours. So, if you open up the app and you had notifications received in 24 hours, you will be in an indirect session.

And then the other thing we add on to that is a notification limit. So, the indirect session can only keep track of a default of 10 notifications at a time and max. So, if you receive 11, it's going to get rid of the oldest, then it keeps track of all the most recent notifications. Here you'll see we have the notifications and then you click on the app icon. I highlight the influence column or indirect session type. You’ll see that it won't count any clicks. A click could never be indirect. And then you'll see the session duration itself is counted at 12.

And then our last session type is unattributed where a user opens up the app with no notifications in the attribution window at all.

Previously, we showed you our workflow for our SDK when it's inside of your app and how we're keeping track of our session focus work. This was previously independent. Now with Outcomes, we realized that it needed to be dependent, where when a new session is started, the focus logic is dependent on the session and restarting. So, if there's no new session, it's going to continuously track focus time itself. You'll see a more in-depth as we move forward.

So, we're going to go back to a blank diagram and we'll start rebuilding it now. So, our session event logic has become refined a little bit. Now, we look at greater than or equal to 30 seconds or less than 30 seconds. In the case where you've been outside of the app for more than 30 seconds and come back into the app, we start a new session. Now, if you left the app for less than 30 seconds then came back, we've resumed the current session. Previously, we weren't really doing this. We weren't really keeping as much attention to it or focused on it. We're just looking at those greater than or equal to 10. So, there are those two events.

Focus Events

[10:13]

The next thing now is the Focus Event stuff, which is a little bit more refined. So, now we're looking at when we should send a focus event. The first case would be, you've left the app and you're outside of the app for more than 30 seconds. We've a background drive that weights that 30 seconds since it would mean a new session. But it will also mean that we send the focus event. Whatever notification information we have would go along with that. And then that's how you would get your session duration, two plus one on the dashboard.

And then the other event is if the app is in an unattributed session and you leave it for more than or equal to 60 seconds, this would be another focus event. But this is similar to the previous one we were making. The new case here is where if you're out of the app for 30 seconds, and when you would come back to the app and be a new session, we want to send that focus event. That's in the case of an unattributed session. And then you'll see it fill in this logic here. So, a little different than before.

So, with app homes on the dashboard, you'll see two defaults. I already showed you them before a little bit. The first one you're going to see is "Clicks", which we keep track of. You'll see above in between these two screenshots, there's going to be a graph on the dashboard, but I just took them out to simplify it a bit. Here you'll see the click-through rate at the top, which is calculated in your percentage, and then you'll see the dashboard with default Outcomes below it, which is our clicks, and that shows a count.

Session Duration

The next thing is the "Session Duration" that we offer. This is for direct and indirect sessions. You’ll see that below with a count and then a sum, the sum being the time in seconds, sum from all the users who have session duration counts to our backend, and then they'd be displayed on our dashboard. Josh will talk about Public Methods.

Josh: Yeah. So, in addition to all the focus time and the number of sessions that Mike was talking about, obviously, we need a way to also be able to provide you the details of basically what happens, what actions the user would take inside your app.

So, we do have three separate methods that I'll cover basically that you can do. So, a very common one is just to have a screen you can set to send Outcomes. So, in this case, you might want to know every single time the user opens like your store in your app. So, if it's like a game or something, maybe it's you in-app purchase store where they have items. So, you just want to know every time they open that.

And then you'll see at the bottom here, if someone clicks on an application and then you send that outcome, why you're in that session, it'll account that as a direct. If it's in that window attribution we were talking about earlier and they just happened to open the app, then we'll count that as influence. So, this is every single time they open it.

We do provide another method which is unique. An example of this would be like for the first purchase that user can only make one first purchase. So, you could basically just set this and call this multiple times, and whichever purchase they happened to make first, we'll track that internal in the SDK so you don't have to worry about that event get duplicating or getting duplicated. So below, you'll see that this should only show up once per user.

Another one is you might want a value to actually go with it. In this case, as an example, you can send a purchase and you can send an amount. So, this could be like a real dollar amount. If it's some games and game currency, you can set that as well, or anything else that's in your app that you can track that has a value associated that you want to basically get summed up. So, it'll show the counts, which is the instances that that was called, and then the sum of all the values or all the weights basically added together here.

Michael: So that wraps up some of the little more technical sides of Outcomes. But now we can show a little demo video that we've created - sending a notification from the dashboard, receiving it on an app yourself, and then showing these Outcomes count themselves on the dashboard.

Josh: I guess we didn't request permission on this video. We'll do that really quickly and start the video here. I'll take a look at the questions here that we have.

Is the 30-second session of the time based on the usage data or is it arbitrary? Is there a way that we can send different file to different people, picture?

[14:58]

Onto the first question first, for the 30 seconds, the 30 seconds is based on a time that we choose. We'd be picked it basically that we felt that was enough time outside of the app that it would be counted as a new session. You should be able to just request it again. So, I go back. Or refresh probably.

Michael: It should be there already.

Josh: Let's try to refresh this. So, with the 30 seconds, the 30-second was just a time that we basically thought that, if the users are outside of the app, they basically context switched into something else and then they'd be coming back. That would be something that we would assume that it's probably going to be something different. So, it is just for right now. It's something we can consider in the future if it's something that you have some suggestions on changing that time.

For the next question, onto this real quick for sending a different picture file to different people, I think you mean like-- yeah. In that case, if you wanted to send a different image to a different person, you would need to send a different notification. So, if you're sending from our API, for example, you're targeting by either a player or maybe your external ID, you just send that one notification with the image to that one recipient and then you'd send another notification to the other recipient with a different picture. So, you just basically send two unifications to get there.

Michael: We'll continue with questions after the demo.

Josh: Yeah. So, I'll have my explain--

Michael: As we're going through the video, I'll explain in a little bit of detail, like what's actually happening.

Outcomes Demo

Right now, we're on our dashboard. We're going to create a new push and we're going to send this to my test device, which you'll see in the next frame. So, we're going to add a title, subtitle and message, the only required fields. And then we're going to do our "send a test device".

What we're going to be trying to do here is trigger a direct session and then show that click and that session duration count, and then send some Outcomes along with it using the public methods. So, I'll send it to Mike, I say plus. So we sent this now and we're going to switch over to the phone. So, now we're on the phone. You'll see we received a notification at the top. We'll go to click this notification and it's going to open up our demo app. Now that we've entered in the demo app, we're going to send some Outcomes. So, let's click Send Outcome.

Now, let's go and send a normal, a unique and an outcome with a value. But we'll start with normal outcome for now. So, we'll just call this 'normal_1'. This can be called anything. I'm just calling normal_1 for simplicity. So, that one sent. Now, I send a unique outcome. Again, it's using generic names, and then value_1, then we'll add a weight to it of 3.4 so now we send our three Outcomes.

So, we're going to go back to the dashboard now and let's look to see about our clicks. And then our three Outcomes went through and they're showing for that notification. So, let's scroll up and let's click the delivery tab. After the delivery tab, let's click on notification. It should be the most recent one. Now, let's look at our data at the bottom. So, the first thing we should notice is that the clicks is one. So, our click went through.

Now let's go and look for our Outcomes. I have a lot of clutter in this list, so sorry. But you'll see our normal_1 or unique_1 and then our value_1 where they all have been counted one indirect. And then we'll look at our value for the value_1. That's 3.4, which is the sum of all of the weights for that outcome.

Now, let's switch back. The goal here is to leave the app for 30 seconds so that we trigger a new session. Now, when we trigger a new session and come back into the app after these 30 seconds, we'd be in an indirect session. So, what we're going to do is open up the app again. Well, first things first, our focus event sent. So, if we go back really quick, you'll see that the session duration count was 1. It changed slides instead of the video.

So, session duration right now, you'll see underneath the clicks is zero right now for direct. But once we leave the app for 30 seconds, we'll send the request for this. Since when we come back into the app, it'd be a new section. So, let's run this one forward. Oh, it's showing outcome still. Now we'll go back to the phone app. Now we'll leave the app and now we'll go back to the dashboard. You'll see that session duration count go to 1. There you go.

So, now what we're going to do is go back into the app on our phone and we're going to now be in an indirect session. Now, we'll send some more Outcomes. We'll do the same exact Outcomes again, normal_1, unique_1 and value_1. For normal, it should act as it did before and add one on top of the original count, but it should be influence this time. Unique shouldn't be going through because we've already sent a unique outcome for this notification that we've received and we've received no new notifications. And then we'll send a different weight for the value_1 just so we can see it.

[20:25]

So, now we go back to the dashboard, we refresh. Now let's take a look at our Outcomes. Our normal_1 got a 1 on influence. Unique doesn't get anything on the influence. It stays as one indirect because we've already done it for this notification. And then our value, sorry, it cut off. There you go. And then you see 2.2 for our value.

So, that's our simple demo video, just showing you how a direct session would work and how an indirect session would work. This is just for one notification. Say, you were in an influence session and you received over the max limit, so over 10 notifications, and then you send an outcome. The outcome would go for all 10 notifications in that indirect session. So, that's a basic little demo of how it works. That's a good little explanation of how we have modified our SDK to work with Outcomes and not just forcing the feature in. now, it'd be a good time for any questions if anyone has any, so we'll just leave it open to you guys.

Josh: Cool. Thanks.

Michael: I appreciate that.

Josh: I guess to touch a little bit back on--

Michael: For all the Android platforms, yes. For all the wrapper SDKs and then the native SDK for Android, it's available. It's only available for iOS Native right now. We're working on the wrappers currently. They should be released in the next week or two.

Josh: Yeah. I guess just to cover one thing with our decision why we're designing this feature. I won't cover this whole flow chart, but just to summarize, because of so many different states that the user can be in or the app could be in, we wanted to flowchart this out and make sure we cover all the cases. So, basically, this is doing all this work and this decision logic that your app would normally have to keep track of, to keep track of Outcomes of what this does in the app. So, we put a lot of time in thinking about, when someone comes to the app, if they received the notification, they opened the app, that got them to come into your app. But if they decided to leave and you get another notification really quick tapping on that, you upgrade them to a new session because direct click would be more important than an indirect or an influence click.

So, there's a number of cases that we have covered and thought about intuitively to fit in and actually give you the proper statistic you're basically looking for.

All right. We'll give another a minute or two here for questions.

Michael: Another really interesting part of this feature, it was just the time tracking, in general, for these notifications. There was a spot where we realized we needed to keep track of our unattributed and attributed sessions completely separate. It got tedious, like integrating our old logic with this new logic with Outcomes. But most outlier cases like this one, where, say, you're in the app for 30 seconds and then you leave the app-- I'm trying to think of the one case. What's the case where you could leave the app and then come back and it's off the track that extra time because it was a new session?

Josh: Oh, I think it was keeping track of the unattributed time versus attributed time.

Michael: I can't think of it.

Josh: Yeah, I think it might have been something around that line. Basically, we don't interrupt the current logic.

Can we white label the SDK?

I would probably get in touch with someone with sales with that. I just don't want to give you the wrong answer since we're not really looped in with all the changes that could potentially be done around that.

[24:48]

Michael: Anybody else? Okay.

Josh: Cool. Yeah. I guess one last thing that came to mind about these Outcomes and that the time we put into it, we basically have a bunch of automated tests around this logic. So, as we're making changes, we're making sure the current logic didn't break in addition to adding this new logic. So, as we go forward, any changes we make to the SDK, we have automated tests in place to make sure that these analytics don't change. Or if they do change, there'll be an intentional thing and we'll make it an announcement or it'd be like another extension to Outcomes or something in the future.

Michael: Any major functionality change.

Josh: Right. Exactly. So, it doesn't look like there are any additional questions. So, I think this will complete our webinar.

Michael: Thank you for joining us.

Josh: Yeah, thanks for joining us, guys and gals. Have a happy Thanksgiving if you're in the US.

Michael: Happy holidays.