ABOUT FAITH: LinkedIn  | DoubleDown Interactive


Hiring for Growth Marketing Specialist  | Hiring for Senior Growth Marketing Manager




ABOUT ROCKETSHIP HQ: Website | LinkedIn  | Twitter | YouTube

KEY HIGHLIGHTS

🔍 The research that went into estimating CPIs for Faith’s latest soft launch

⬆️ The impact of higher CPIs on the soft launch process

🧮 The disruption of measurement post ATT impacted soft launches

🗝 Why launching post ATT meant having fewer partners

📤 How to approach conversion values

📱 The challenges with conversion values for an IAP based app

💎 Why custom conversion values worked better

🧱 How conversion values are architected

🎞 How to approach optimization for social channels

📊 How a BI team can significantly help improve measurement

📝 Comparing between measurement techniques

🎢 What was different during the recent soft launch vs. pre ATT

☎️ What were some conversations within teams like when soft launch metrics were so different from predicted or projected?

🔖 Faith’s advice for marketers for a successful post-ATT launch.

KEY QUOTES

The stark differences between estimated and actual CPIs

I would say the estimated CPIs we had were maybe 25% of what actual CPI is ended up being, even when you’re looking at getting those initial installs just to look at retention, not necessarily specifically getting into high value revenue sources.

What led to the discovery that Android was becoming more expensive to test in

So there was a spike in Android spending. The other thing we discovered was Android was becoming more expensive than iOS. We had an RPG game and we thought that we would do really well in the Android market versus iOS, because that’s the rational thought process around RPG, given their demographic.

What product teams had to re-assess post initial results

How are we going to view the game and the target audience considering that the value of the player doesn’t necessarily go up?Just because CPIs increase doesn’t mean that the player value increases. And so I think they had to go back then and look at their game and look at what sort of core loops did they have in their game? And how are they monetizing and taking into account their LTV curves and say, was their initial planning for LTV curves and demographics going to work in this new marketplace?  

Approaching measurement with all the realities of ATT

One was for iOS to limit the partners, maybe more than we would have in the past. As you mentioned, measurements have been a bit disrupted and so the fewer partners that you have, the more confidence you’re going to have in that information. We had conversations with a lot of partners who were very interested in working with us to soft launch the game. And we had to let them know that we really were being very, very conservative about how we choose which limited partners we want to deal with in order to make sure we can get the leading indicators on the game. 

How are custom conversion values important

The reason we go with custom conversion values is we need to look at more than one indicator in a game. And in part because IAPs are a lot less frequent than almost anything else in a game. So you can’t just go with a revenue event. Obviously, the best indicator of somebody’s going to purchase is if they purchase, but those purchases are a very small percentage. And when you’re looking at SKAN, where you’re already getting that percentage of actual data and not the full actual data, it sort of compounds.

Understanding custom conversion values

Let’s say we have the lowest level of conversion value might be somebody who comes in day one, and completes a level. And then the next one might be they come back on day two and complete a level and maybe they complete five levels. So that’s going to be a higher conversion value than somebody who only completes one level. And then you start layering the things – maybe they do tutorials or, unlock mini games, and then eventually you see that they’ve done these things. So that can really tell our partners around the conversions that you’re looking for anything higher than x.

Tracking conversion values for social channels

We have to look at what we started doing by making sure we have a weekly look back, maybe looking back for the last four weeks. And letting that data mature instead of saying “okay, we’ll really focus on what’s going on this week. Okay, well, how did it get better? Two weeks ago, and three weeks ago?” We also look at blended ROAS now. How is the product as a whole doing? And you know, if as a whole it’s doing okay, and I think this is easier because we have a product that’s 10 years old, as well as one that’s three months old, to try to balance those out.

Measurement techniques that were tested and approved

We have evaluated that with some of our partners and said to them, we’re willing to try probabilistic or deterministic with you, but you have to have a backup plan for SKAN. You have to be using this data, we don’t want to have essentially all of our eggs in one type of attribution basket. And if something shakes up the industry again, you have to have a plan B that you’ve been working on that you’re ready to roll out.

What did the conversations look like with the soft launch results post ATT

I think the biggest impact out of that was really talking about what does your LTV curve look like? And our conversation with them too was then, what does that do? Like with these CPIs, what does that do to our Dx ROAS goals? And what we’ve seen, unfortunately, is there’s more pressure on the front end to get higher D7 ROAS than there might have been a year or two ago. And so we’ve had to say “Okay, well, if this is the marketing cost, here’s what we really need to fit this LTV model.” And trying to temper it and say, “Okay, well, realistically, this is the type of traffic and this is the type of performance we can drive.” 

Preparing for a better soft launch with existing data

We’ve also talked about things like shorten testing cycles, instead of doing a fully developed game, is there a new style we can bring out where we can do a smaller version of the game and make sure that there’s interest around it so that we can cut down the marketing costs of soft launch by not doing full blown launches? On games, we have no data in terms of whether or not the user likes the theme and the mechanic and the core loops. Looking forward, now that we have all this information, how do we do better for the next soft launch? And how does the development side of teams and the design studio make a better testing process for the games?

FULL TRANSCRIPT BELOW

Shamanth  

I’m very excited to welcome Faith Price to the Mobile User Acquisition Show. Faith, welcome to the show.

Faith Price  

Thanks, I really appreciate you inviting me. 

Shamanth  

We are very thrilled to have you. We connected over a year ago and even crossed paths digitally. I have a lot of respect for everything you’ve done over the years – and I’m very honored to have you today. 

The theme of our conversation today is going to be something you’ve done a lot of – you worked on over 20 soft launches throughout your career, and you clearly see so much of what goes into launching hit games. You’ve also been a part of a launch after Apple’s ATT came out. 

To start off, what did you see differently with the post ATT launch? Clearly you were looking at CPIs, but what did you see the CPI trends as being? And why do you think the CPIs were the way they were?

Faith Price  

I want to preface this by saying, our team went into this aware that we would be launching in an unprecedented time that we didn’t have a lot of data on. That being said, we did try to do as much research as possible first, and this was also a new genre since Doubledown Interactive is primarily known for casino titles and this was a zombie apocalypse sort of game. 

So we went into it and did research ahead of time around where we should be launching, which soft launch countries should we be looking at? We did a lot of reviews of what other similar titles were in soft launch. Where were they in the soft launch? How long have they been in soft launch. And then we went to the partners that we work with for media, and asked them to provide information on estimated CPIs for our genre, for our type of title for those geos, as well as, what were the top 10 videos in general for those types of titles. So we tried to get as much information ahead of time as possible. 

We could make some educated assumptions about what the CPI should be and where we should be because soft launch countries can change over time. Especially in tier 3 and tier 2, there’s a lot of movement around technology and more people gaining access to it. We did want to make sure we were going in the right place. And what we found once we actually went out into soft launch was, the estimated CPIs from our partners was very much understated. 

Granted, they were looking at their historical data, even just looking back the previous three months, because we really had to soft launch about a month after there started to be the wide adoption of ATT so it was still relatively new.

So for the most part, everything was just a lot more expensive than we had initially planned for it to be. I think it became especially predominant when we were looking at tier 1 geos. The impact of ATT was even more pronounced there. So we had to evaluate what we launched and see what was going on, because up until 2020, I would say CPIs had gone up every year. Our media partners even said, we expect CPAs to go up a certain amount every year. But just the last few years have been a little bit of a wild ride. 

Shamanth  

That’s crazy. And you’re right in that even if you look back three months before you launched, there was just probably no way to estimate what is going to happen. Thank you for actually describing the process of evaluating how you reached out to partners to get these estimates. Because I think it may be obvious to you and me to some extent because I’ve laaunched a couple of games, but I do think there are a lot of people to whom it may not be very obvious as to what happens before it’s live. So thank you for actually describing that. 

So you guys launch and clearly CPIs are way above what was targeted and budgeted. What is the impact of this higher than expected CPI on the rest of the soft launch process?

Faith Price  

I think one of the impacts of it was the product team was more invested in getting information on whether or not this is an investable game. Because I had initially come out with what I thought was a reasonable soft launch budget, to give guidance to the executive team, then a couple weeks into it we were revising that considerably to say, we thought it was going to be one thing but no, it’s quite a bit higher. And the other thing I should mention too, in preparation for ATT, I think a lot of game companies started shifting budget over from iOS to Android. 

I can actually see that in the other third party sources that I looked at in terms of like, where our competitors were ranking on some videos, and some platforms three to six months before ATT as compared to now.

So there was a spike in Android spending. The other thing we discovered was Android was becoming more expensive than iOS. We had an RPG game and we thought that we would do really well in the Android market versus iOS, because that’s the rational thought process around RPG, given their demographic.

The challenge then was, we had to talk with the design team about how Android is a lot more expensive. 

So how are we going to view the game and the target audience considering that the value of the player doesn’t necessarily go up?Just because CPIs increase doesn’t mean that the player value increases. And so I think they had to go back then and look at their game and look at what sort of core loops did they have in their game? And how are they monetizing and taking into account their LTV curves and say, was their initial planning for LTV curves and demographics going to work in this new marketplace?  

Shamanth  

Yeah, so many unknowns! Something you said earlier on struck me as important because you said you had to decide whether the game was investable or not. And all of a sudden it was so much more expensive to have to decide that. Because ultimately, that is the goal of the soft launch, right – should we move forward or not? 

I know we talked about the CPIs going up. But it wasn’t just the pricing that was shaken up by ATT, it was also the entirety of measurement right? And in a soft launch you’re looking at crash rates and also how the retention is shaping up. How’s your monetization shaping up? Based on my experience, this is completely disrupted. How did you approach measurement during soft launch with all of the realities of ATT?

Faith Price  

Our decision was in two parts.

One was for iOS to limit the partners, maybe more than we would have in the past. As you mentioned, measurements have been a bit disrupted and so the fewer partners that you have, the more confidence you’re going to have in that information. We had conversations with a lot of partners who were very interested in working with us to soft launch the game. And we had to let them know that we really were being very, very conservative about how we choose which limited partners we want to deal with in order to make sure we can get the leading indicators on the game. 

That was one of the challenges. And I think for the other one, we really had to take a look at – for this specific type of game, how are conversion values going to work? That’s your other leading indicator on the game. 

Shamanth  

So it’s just hard to manage 15-20 partners like we’ve done in the past with ATT and SKAN just because you can’t isolate the impact of each and I think that certainly makes sense. We’d love to hear more about conversion values. How did you guys approach conversion values? What are some of the criteria you used? 

Could you tell us more?

Faith Price  

I think the biggest challenge with SKAN is, they’ve tried to make it a one size fits all sort of environment and that isn’t necessarily conducive to every type of game. For us with conversion values, part of the challenge is you need enough data to determine what events in the game are actually indicators of revenue. For the initial soft launch, and for most of it, we were just using one of the preset revenue events that our MMP provided for conversion value. And we used that to help build up enough data to be able to determine once we hit worldwide launch – what do we really want our conversion values to be? Because we work with custom conversion values, that’s the only way for the type of games we do to get actual valuable information on the users. 

All of our games are mostly IAP based. I think if you do a game where you’re heavily reliant on ad monetization, then you’re going to have a different experience, potentially, what should your conversion values be? That’s where you really need to have a strong data analysis team that can go in and look at the data and look at all the different types of events and make those considerations on what your conversion value schema should be based off of. In soft launch, we had to recognize that we weren’t necessarily going to have a lot of good data to then build in time, before worldwide launch to actually change the conversion value schema, because that’s one of the things that you don’t think about, the extra work around SKAN is for some partners, once you change SKAN, you have to wait up to three days for your new campaign to actually be able to launch, we work with a fairly short and abbreviated launch period. 

So there were a lot of extra conversations between engineering and development and the BI team and us to try to make sure that everything was aligned for worldwide launch for the conversion values. But really, it was a data gathering time and they were able to produce something in advance of worldwide launch, which is our current schema for conversion.  

Shamanth  

Yeah, and I think it’s especially challenging for IAP focused games, just because there’s less of a revenue correlated signal. I’m curious, can you help us understand what you mean by custom conversion value? Could you talk about what that means? And even if that’s a high level schematic understanding,

Faith Price  

The reason we go with custom conversion values is we need to look at more than one indicator in a game. And in part because IAPs are a lot less frequent than almost anything else in a game. So you can’t just go with a revenue event. Obviously, the best indicator of somebody’s going to purchase is if they purchase, but those purchases are a very small percentage. And when you’re looking at SKAN, where you’re already getting that percentage of actual data and not the full actual data, it sort of compounds.

So what we have done at sort of a high level is taken other indicators in the game, whether it’s a retention indicator, or an activity in the game indicator, like a lot of people, for example, would have levels in a game, right? So we’re looking at things the user does, it systematically moves them through the game, and then trying to divide those up into relevant bits that we can put an indicator on, and then also pair some of the revenue and purchase events and what type of purchase event for example. So somebody who maybe makes a purchase might not be as valuable, or that might be an indicator that they’re not going to spend as much as somebody who makes a higher price purchase. 

So trying to put multiple things into a single conversion value, to hopefully get better indicators. Again, we’re talking small percentages because they have to do something a certain number of times even though Apple is or even though the SKAN values you can have them continue for up to 64 I think 63 days. For us that is way too long of a time period to be able to do anything relevant, right? So when’s the cutoff point? When do we say “Stop sending me conversion values” . At that point the traffic that was actually responsible for those conversion values and the market conditions, maybe with that specific was responsible for those conversion values is now lost. 

That was the other thing they had to figure out was when do you stop and say, “Okay, anything after this, we can’t do anything useful with.”

Shamanth  

Yeah, you have to find that sweet spot. Because I know a number of folks we’ve worked with, who said “We’re just going to take day zero or day one, depending on how you define it. Because anything past that, we just don’t want to trust that.” 

I’ve certainly seen a gradual shift towards expanding that time window. I understand what you said about custom conversion values. So what you’re saying is, you’re looking at a combination of events. So a user finished level 30, watched five ads and added a friend, and if that complex event correlates to revenue, that’s bit one. Is that roughly how you’re looking at it?

Faith Price  

That’s very similar. But

Let’s say we have the lowest level of conversion value might be somebody who comes in day one, and completes a level. And then the next one might be they come back on day two and complete a level and maybe they complete five levels. So that’s going to be a higher conversion value than somebody who only completes one level. And then you start layering the things – maybe they do tutorials or, unlock mini games, and then eventually you see that they’ve done these things. So that can really tell our partners around the conversions that you’re looking for anything higher than x.

Here are the numbers that we have, here are the conversion values for people who do things in the game that aren’t necessarily spending. And then here’s the higher set for people who do things in the game and do spend, and really anything above this number is an indicator. 

Shamanth  

That makes sense. They have a clear target to shoot for. With social channels, Facebook, Google, Tiktok, it’s not nearly as straightforward. You don’t see cost per conversion value as easily. I’m curious again, to the extent you’re comfortable sharing how to approach optimization or tracking conversion value for the social channels.

Faith Price  

I feel like the world of attribution is still really in flux for social, search, etc. And I think we’re seeing that even with Facebook’s relatively new move at the end of October for the Android attribution and the big changes they made there, it really becomes specific to the partner as to how much leeway do you give them? We have specific reporting that’s built to consider any data that comes in via IAP, any data that comes in via SKAN and then do an amalgamation. 

UA has become a little bit more fuzzy. Here’s this partner’s data. And we know historically that they tend to get a lot more reliable SKAN data. Maybe for some of the search and social, we still have some questions. So we’re not sure that it’s as reliable. So depending on the partner, maybe give them a little bit more leeway when we’re looking at optimization for a week where the numbers are down. The performance doesn’t look like it dropped suddenly. But the other thing is also, we do and I think most UA teams do a 7 day look back. And the data is just really delayed for some partners, especially social. 

So again,

We have to look at what we started doing by making sure we have a weekly look back, maybe looking back for the last four weeks. And letting that data mature instead of saying “okay, we’ll really focus on what’s going on this week. Okay, well, how did it get better? Two weeks ago, and three weeks ago?” We also look at blended ROAS now. How is the product as a whole doing? And you know, if as a whole it’s doing okay, and I think this is easier because we have a product that’s 10 years old, as well as one that’s three months old, to try to balance those out.

Shamanth  

Sure. Yeah. And as you said, all of UA has become a lot more fuzzy. And I think also the way you describe your process, I would imagine the off the shelf MMP product in itself isn’t enough. It sounds like you guys have to lean on a lot of in house BI capabilities to do that. 

Faith Price  

We were set up because we always have, they’ve always looked at the data as it’s come out of the MMP and made some assumptions. We’re taking a deeper look into it, especially because we have a 10 year old product and there is so much reengagement in those cohorts. We already had a fairly robust internal reporting system setup so they then had to apply it to the new title. So I would say, if we didn’t have that, it would certainly be challenging, I would certainly have more questions about that data, which is really unfortunate. I did a little bit of a stint in traditional marketing before I jumped sort of feet first into digital marketing, and affiliate marketing. 

One of the things I loved is there was a lot more lack of ambiguity around results, you could follow that path. And we don’t necessarily care who that individual user is, we care more that it’s a piece of data that we can say, this particular data point says they saw this ad, this ad was X amount persuasive, which led to this result. Now, I do feel like there are some interesting things coming down the pipeline, one of which just launched with Apple – the custom product pages. I think if I was doing a soft launch now, I would certainly consider for iOS, at least having every partner go do a custom product page. 

Because I think, for one thing, obviously, you can tie creative in through the entire user experience, the user journey, but also you would have a better understanding of this high level, top of the funnel indicators. So that might provide a little more data. But one of the things you said was actually interesting. You were talking about the breadth of partners, and I will say over the past, even leading up to ATT, we’ve really looked at the partner, the potential partner base and media base, and we’re a lot more strict around, who do we agree to test with? Because, testing is just expensive. And in terms of resources, it’s the monetary factor and I feel like there’s a lot of overlap, too. So we really challenge the partners that we work with, to show value. Like if everybody is buying off of the same ad networks, we don’t want to be competing with ourselves. And so I think what I’ve seen from when I first started in 2014, is just a streamlining of partners that we’re willing to work with.

Shamanth  

If they’re giving you a 50-100 install today, it’s oftentimes just not worth the time and resources. I know we talked about some of the ways you looked at measurement with SKAN. Something that’s been talked about a fair amount post ATT is non-deterministic measurement and this has been proposed and talked about as a partial antidote to measurement challenges. Some of the things I’ve heard, and to some extent tested myself have been, more probabilistic attribution, MMM based analysis, incrementality based analysis for measurement. So are these sorts of techniques that you tested, evaluated, adopted, found effective, what has your experience been?

Faith Price  

We’ve worked with a variety of partners who are trying a variety of things. And I will say, for the time being, that we have seen some partners who work with more probabilistic attribution be successful. The way we look at it is, we’re interested in trying  a lot of different things and seeing what sticks. We have not, I would say, built out our own probabilistic modeling. 

Given our particular strategy, I don’t know that that’s something that we’re necessarily going to be looking into. I’ve chatted with a couple of the larger game companies who have a full portfolio of titles who have been looking into what it would take to do a lot more of that internally.

We have evaluated that with some of our partners and said to them, we’re willing to try probabilistic or deterministic with you, but you have to have a backup plan for SKAN. You have to be using this data, we don’t want to have essentially all of our eggs in one type of attribution basket. And if something shakes up the industry again, you have to have a plan B that you’ve been working on that you’re ready to roll out.

Shamanth  

Yeah. And it’s hard to know when Apple might just say, probabilistic is not going to fly. Because again, a lot of people will just call it fingerprinting. And depending on who you ask, and how they define it, it could just all change very soon. 

Earlier on, you talked about how the conversations you started to have with the game teams about when you saw the first data come back from the soft launch. So can you tell us a bit more about what it was like to have those conversations, perhaps with your finance teams, with your game teams? I’ve had my share of debates and conversations with finance teams on what’s reasonable and what’s realistic. What were some of those conversations? Like when you went back to the teams with the results about elevated CPIs and the measurement issues? 

Faith Price  

One of the things that we did for this soft launch that was different from other soft launches I’ve been in is, we had a lot more regular conversations with the larger marketing groups starting before soft launch. So starting a couple months before launch, we had regular scheduled meetings, all throughout soft launch and even after soft launch and worldwide. So as we were presenting results, these conversations just became part of them. So they had all of this in real time that these costs were going up. 

And

I think the biggest impact out of that was really talking about what does your LTV curve look like? And our conversation with them too was then, what does that do? Like with these CPIs, what does that do to our Dx ROAS goals? And what we’ve seen, unfortunately, is there’s more pressure on the front end to get higher D7 ROAS than there might have been a year or two ago. And so we’ve had to say “Okay, well, if this is the marketing cost, here’s what we really need to fit this LTV model.” And trying to temper it and say, “Okay, well, realistically, this is the type of traffic and this is the type of performance we can drive.” 

We have multiple game development studios under Doubledown and with our sister company Double U Games, how does that inform the next games coming down the line? And do we look at multiple monetization strategies? Like I said, we’re primarily IAP based, but do we need to think about focusing on some in-game advertising to help make sure that we’re monetizing to the fullest extent to cover the fact that our CPIs and the marketing costs have gone up. I also think it’s potentially a higher bar for the game design itself because so few games make it through soft launch to worldwide launch. I’ve even seen some that have made it through soft launch and into worldwide. And then again, they just don’t last more than a year or so. 

We’ve also talked about things like shorten testing cycles, instead of doing a fully developed game, is there a new style we can bring out where we can do a smaller version of the game and make sure that there’s interest around it so that we can cut down the marketing costs of soft launch by not doing full blown launches? On games, we have no data in terms of whether or not the user likes the theme and the mechanic and the core loops. Looking forward, now that we have all this information, how do we do better for the next soft launch? And how does the development side of teams and the design studio make a better testing process for the games?

Shamanth  

Yeah, and a lot of this is testing and a learning experience, because I don’t think anybody really envisioned what was going to happen. You talked about learnings going forward. I’m curious if there are any specific takeaways or learnings that you would point to as you would recommend for teams that would want to approach soft launches, and testing going forward?

Faith Price  

I would say a couple things. One is, do your research. Now that ATT has been around, there’s a lot more data. Make sure you get a more accurate estimate on what your marketing cost is going to be. Have those conversations in advance with the finance team and the design team. I will say we didn’t do it in soft launch, because there’s a lot of functionality that continues to be built into the game. But if we had promo codes and things like that, I think those would have been super useful indicators as well for specific traffic sources. So when we do our next soft launch, what are the things that we can do around trying to help get more reliable information? If you’re doing soft launch now, make sure you know where you should be going to have better, more accurate estimates of the CPIs. And then try to put things in place to help get better information for specific traffic sources. Mostly, it’s to throw out the old book and write a new one.

Shamanth  

Yeah, it’s a whole new playbook that we have to use. And certainly you explored what  the components of the new playbook are with your first hand experience. So thank you for sharing with us what happens behind the scenes. This has been very, very instructive and is perhaps a good place for us to start to wrap up. But before we do that, could you tell folks how they can find out more about you and everything you do?

Faith Price  

Sure, I am running around on LinkedIn as most people are. But I’m also on a couple of the slack forums like the Mobile Dev forum and the Liftoff forum. I think with everything going on, it’s really been even more key to have conversations with people all around the globe in terms of marketing. So I tend to frequent those forums and try to participate.

Shamanth  

We will link to your LinkedIn so people can find you. For now, this is a good place for us to wrap it. Thank you so much for being a guest on the show.

Faith Price  

Thank you, I really appreciate it. I will say one final thing, which is a little plug for my team. We are actually looking at two open roles right now, for experienced UA buyers. So if there’s anybody in your audience who is looking for their next opportunity, we can have that conversation.

Shamanth  

Excellent. We will link to that in the email newsletter which goes out to about 3000 folks in growth and marketing. And it will certainly be in the links on the show notes as well. Wonderful Faith, so I will let you carry on. Thank you so much.

Faith Price  

Thanks so much. It was great chatting with you

A REQUEST BEFORE YOU GO

I have a very important favor to ask, which as those of you who know me know I don’t do often. If you get any pleasure or inspiration from this episode, could you PLEASE leave a review on your favorite podcasting platform – be it iTunes, Overcast, Spotify, Google Podcasts or wherever you get your podcast fix. This podcast is very much a labor of love – and each episode takes many many hours to put together. When you write a review, it will not only be a great deal of encouragement to us, but it will also support getting the word out about the Mobile User Acquisition Show.

Constructive criticism and suggestions for improvement are welcome, whether on podcasting platforms – or by email to shamanth at rocketshiphq.com. We read all reviews & I want to make this podcast better.

Thank you – and I look forward to seeing you with the next episode!