ABOUT: LinkedIn  | Twitter | Website

Matej’s course on Mobile Dev Memo Academy: Soft-launch strategy for mobile games

ABOUT ROCKETSHIP HQ: Website | LinkedIn  | Twitter | YouTube

KEY HIGHLIGHTS

📲 What goes into the technical stage of a soft launch.

👀 What metrics to look for in the technical stage of a soft launch.

⏳ What metrics to look for in the retention stage of a soft launch.

🤩 Why it’s preferable to soft launch on Google Play before iOS.

💸 What metrics to look for in the monetization stage of a soft launch.

🔎 What volume of payers to aim for in the monetization stage.

✅ Why it’s important to do the retention test before the monetization test.

💭 Does this way of thinking about soft launches change for subscription apps or ad monetized apps?

😮 The common mistakes that developers make with soft launches.

⏱️ Why it’s important to ideally look at a six month time frame for soft launches.

🤦‍♂️ One outcome of a soft launch might be that you have to kill your app.

KEY QUOTES

The importance of a smooth experience

(In the technical phase of a soft launch), you’re looking at mostly a crash rate or an ANR rate – everything that blocks people from playing the game. We just need to be 100% sure that people can play and play well. 

Putting onboarding under a microscope

(In the retention stage of a soft launch), it is really important to measure the tutorial completion rate, besides the classic retention rate 1d, 7d & 28d –  the onboarding flow of the user and measuring each step from the flow to check if there is not any problem which might cause people to drop out.

Quick results on Google Play

I started to do soft launches on Google Play. Why? Because, I don’t want to wait two days for Apple to review the game. Basically, the game is live on Google Play in a couple of hours.

Monetization is worth the expense

(In the monetization stage of a soft launch), you would need 10,000 users for a month or 20,000 which can be obviously pretty expensive. But this is the time you can prepare for the global launch and then you are budgeting or trying to allocate some budget for a global launch, so you need to be, not 100% sure, but 200% sure, even more. So you don’t want to waste the money. That’s why obviously the monetization stage will be the most expensive, but you can definitely benefit from this stage.

Soft launches are just the beginning

One mistake that people make is they think that soft launch can be done in one month. Because they think: we did everything needed in development. And now we just launch the game in a couple of countries and do some iterations. And then the game is ready for global launch in one month. 

But well, definitely not. I mean, you still need to be in this soft launch for three to six months to see how the players or the cohorts mature. There’s still a lot of things you need to test like creative testing, testing the new UA channels—there’s tonnes of stuff you need to do before going global. Yeah, and you just can’t do it in one month. That just can’t happen.

Kill your darlings

We need to speak about the other side, not only going global but also you have to sometimes kill the game. Because if you don’t see your KPIs improving, and you’re stagnating on the same numbers, then it’s just better to kill the game right away, because you are missing out on the opportunity of creating something better and something else.

FULL TRANSCRIPT BELOW:

Shamanth: I’m very excited to welcome Matej Lancaric to the Mobile User Acquisition Show. Matej, welcome and welcome back. 

Matej: Thank you very much for having me again. I really enjoyed the last episode when we talked about the hypercasual games. I’m even more excited now. 

Shamanth: Indeed, quite a few people have referenced your hyper casual episode, some of our clients, some on our team as well. So definitely excited to have you back. Also, and we are going to link to this: Matej is going to be teaching a course on the Mobile Dev Memo Academy, folks can check that out on mdm.academy. We’ll link to that in the show notes and the transcript just as well. And certainly, in today’s episode, we’re going to talk about what is one of Matej’s specialties, which is also the topic of his Mobile Dev Demo Academy – soft launches, really, which can be a whole universe in itself. 

Today, we’re going to talk about the three stages of a soft launch. These are the three key steps of a soft launch that I think every app developer should nail to really go global. So Matej, to start off, can you tell us what the three stages are?

Matej: Yeah, of course. First of all, I think that the soft launch is super important to every developer. And so basically, I’m trying to divide the soft launch, it’s like the common knowledge  – to these three stages, as you mentioned, and it’s the technical phase, then like the retention stage, and then finally the monetization stage.

Shamanth: Yeah. Let’s dive into each of these three stages in sequence. The technical phase: I would imagine is somewhat evident, but can you speak to what happens during the technical phase of a soft launch?

Matej: Of course. The technical phase is the first thing that you do after the game is done in development. You want to try and test the game’s stability and the whole tracking before you start to spend a lot of money in the next stages – but then like this stage is all about identifying the issues, bugs, testing the infrastructure, all the tools like data tracking, attribution, everything from the technical side of the game, and then finding and fixing those critical bugs that might occur while setting up the attribution platform, or Facebook SDK or any other SDK you might have in the game. 

Basically, in this stage, you don’t need a lot of budgets here because you just can test this with a small $50 or $100 budget, because we are acquiring players from tier 4 countries. Why tier 4 and what does it mean? It’s like the Philippines or Mexico or any other country that has very low CPIs because you want to spend the budget effectively and you don’t want to test the technicalities in the UK or any other tier one country because it would be expensive.

Shamanth: Sure, right. And it certainly makes sense. You are testing the waters, you’re making sure the game is stable. And you’re sending very cheap traffic because all you want to make sure to do is that the game doesn’t break. What are some of the key metrics to look for during the stage? And, when do you say during the technical testing phase that okay, everything is good, we could go to the next stage – what are the criteria you look for?

Matej: So basically,

(In the technical phase of a soft launch), you’re looking at mostly a crash rate or an ANR rate – everything that blocks people from playing the game. We just need to be 100% sure that people can play and play well. 

And then if we see like all the installs we should in an attribution platform, or Facebook or any other UA channel you use for this – though I would suggest going for Facebook, because it’s super convenient. Anyway, I use it as well. And so basically checking if what we see in Facebook matches with the attribution platform and matches with the store itself. So, these would be like the main points to look for.

Shamanth: So on the one hand, you are looking at data integrity, which is if my data was actually captured accurately, in Facebook, in attribution, then you’re also looking at technical stability, you’re looking at crashes and ANRs. Are there benchmarks – and, for those of our listeners who aren’t aware, can you tell us what is ANR? And also tell us what sort of benchmarks might they look for in crashes and ANRs?

Matej: So ANR is App Not Responding. I’m checking this in the Google Play console, because Google Play has vitals where you can find all the benchmarks. ANR I think it’s like, under 1% or under 0.5%. And then crash rate benchmark thresholds should be around 1%. You can find in the vitals in Google Play console, I always look when I see crash rate spikes or so.

Shamanth: Sure. On iOS, how might they find this out?

Matej: Actually, there is a very similar thing in the iOS iTunes console, when you check it’s like a very similar thing.

Shamanth: Fair enough. And I imagine there are third parties also that allow you to track these things now.

Matej: Yeah.

Shamanth: And, yeah, for folks who are curious, I’ve used HockeyApp. And I believe Crashlytics lets you track these as well. And, so right, you’re like, okay, my crash rates are under 1%, under half a percent – we know it’s somewhat technically stable. We know the installs are lining up. Okay, what’s the next stage? What are you looking at in this stage?

Matej: Yeah. So basically from the technical stage you are going smoothly to the retention stage, and then adding more countries, obviously not tier-4 countries but then, tier-2 or tier-1 countries, mostly tier-2 to test the retention and not spend a lot of money with high CPIs. So basically, here I’m looking for countries like the Netherlands for example, Denmark or any other European countries.

(In the retention stage of a soft launch), it is really important to measure the tutorial completion rate, besides the classic retention rate 1d, 7d & 28d –  the onboarding flow of the user and measuring each step from the flow to check if there is not any problem which might cause people to drop out.

So basically you have the tutorial then you’re checking also if there is not any problem in tutorial flow, and then maybe also started testing tutorials. Like AB testing which tutorial could bring better results, better retention, which of those tutorials could potentially explain what this game is about in a better way. So these are the main focuses also from the product side, and obviously, in this stage, we should be thinking about starting to test creatives and maybe adding the iOS Build. 

So basically, I would have even mentioned before,

I started to do soft launches on Google Play. Why? Because, I don’t want to wait two days for Apple to review the game. Basically, the game is live on Google Play in a couple of hours.

So, then, that’s why I mentioned that I add the iOS build later on, it is very much appreciated. And you know, you want to compare the results between iOS and Google Play.

Shamanth: So, you start with iOS later on after Android. And in the retention test phase, you’re essentially seeing, are the users getting the user experience, are the users playing, is there a problem with the user flow. Because if you know if the users understand what the game is about, then their retention is going to be strong, it is as much about the gameplay, rather than the technology at this point.

Matej: I mean, obviously, you’re still checking if there is not something wrong with technicalities. But yeah, you’re mostly focusing on the product and the flow of the players.

Shamanth: Sure, are there benchmarks that folks should look for? And I imagine this would be different by genre, you know, casino games would have different retention compared to a hyper casual, but other broad benchmarks that people could look at?

Matej: Yeah, there are these benchmarks, they’re all in the gaming industry like 40-20-10, it’s like 40 day 1, 20 day 7 & day 30 – 10%. I mean, as you said, different genres, different benchmarks, I would say. I mean, if you have a 40 day one, it’s still very, very solid, d1 retention. But if you can get like 60 day one in, for example hyper casual, then it’s perfect. But I mean, like for any other genre that you can get 50% day 1, that’s pretty solid.

Shamanth: Yeah, certainly certainly, right. So, let’s assume an app clears stage one and stage two. Stage one that crash rates, everything is good, the data’s stable. Stage two, they’re hitting 40-20-10, or whatever their benchmark is. They move on to the third stage. What’s the third stage?

Matej: Yeah, so the third stage would be the monetization stage and the thing is that the retention stage and the monetization stage could work or could happen simultaneously because it depends on the capacity of the team. Because, obviously if your team is small and you can focus only on a couple of things, then starting with retention is somehow easier. And then if you are still okay with the retention that you were able to achieve, then you are moving up to the monetization stage. 

And then like, what are we measuring in the monetization stage. So basically, like all the monetization KPIs are like conversion rate to payment, average revenue per user, average revenue per DAU, LTV; basically in this stage, you’re like starting to focus more on the ROAS campaigns and purchase optimised campaigns to have a better picture about the monetization KPIs – and then the goal here in this stage could be like achieving the LTV higher than the CPI obviously or certain percentage of ROAS at day 7, day 28, day 30, depends on the benchmarks themselves. And it’s very much important to be in constant discussion with the product team and the analytics team to be able to say what’s going on with the game and with the marketing and everything.

Shamanth: Yeah, yeah. And, you know, when you’re testing monetization, the users are going to be more expensive. Your marketing budget has to be larger. And also, you have to be mindful of the conversion rate, because if you have to get one payer, you need 100 installers, so that adds to the cost. So what volume of users should anybody aim for during the monetization stage?

Matej: Yeah, so basically, this is why I mentioned the constant discussion with the analytics team because they would set some targets for you. For example, we need 500 paying users and give or take, because those amount of users will give us statistically significant numbers that we can make a better decision. So, you know, it can be 500, it can be more depending on the discussions and what the analytics can say about that. But then, again, if as you mentioned that if you have different conversion to payment, then you need even more because we’re still talking about the paying user. 

So, if you have a certain percentage in terms of conversion rate payment,

(In the monetization stage of a soft launch), you would need 10,000 users for a month or 20,000 which can be obviously pretty expensive. But this is the time you can prepare for the global launch and then you are budgeting or trying to allocate some budget for a global launch, so you need to be, not 100% sure, but 200% sure, even more. So you don’t want to waste the money. That’s why obviously the monetization stage will be the most expensive, but you can definitely benefit from this stage.

Shamanth: Yeah. And, something I also like to point out to people who asked about the expense is, if you will know within one week if it’s really bad, right? You don’t need 500 payers to know it’s really bad. If on your first 30 payers, you’re very, very ROI negative, you don’t need to wait for 500. So I think that’s one way they can look at the expense that is involved in the monetization test.

Matej: Of course, but you need to have this budget allocated before. Then, you know what to complete in P&Ls, for the team, for the game and everything. 

Shamanth: Absolutely, absolutely. This may be somewhat obvious, but I’m curious why we should do the retention test before the monetization test and why not the other way around. 

Matej: As I mentioned during the retention stage, you’re still like focusing on the user flow and the things that have higher impact on the retention because then you know from the beginning of the game and early stage of the player journey. The monetization is a little bit later in the stage. So, it’s much easier for me and for the team to focus on the early stage of the player funnel. And then if that’s in good shape, then obviously moving to the monetization stage will be the next step.

Shamanth: So it’s almost like you check this box before you actually invest more into the game itself.

Matej: Of course, because you’re acquiring users. And if you have a problem with retention, then obviously the higher funnel metrics, you are already losing there. So you will lose even more on the down side on the payers.

Shamanth: Right? So you’re almost looking at it as like: let’s test up funnel metrics before we test down funnel metrics.

Matej: Yeah, exactly.

Shamanth: Yeah and I know you said you wanted to get 500 paying users for the monetization stage. What might be the kind of user volume you should look for at the technical stage and the retention stage?

Matej: Yes basically for the technical stage, 100 users that’s pretty much it, if everything is okay. The first round of testing and the technical test is 100 users, then if everything is fine then moving on. If there is obviously some problem, then you need to acquire more, but it’s not like in the 1000s. It’s like in hundreds and it’s really like 50 or 100 bucks.

 Retention stage depends on how much you want to AB test, and what you want to AB test. And we’ll do maybe a AB testing tutorial, then you need about like 4000 or a lot more thousand players than in the technical stage. So, I’m always looking for at least 200 daily new users just to have a better idea with a statistically significant number of players.

Shamanth: Yeah. And I imagine if you have 200 per day, you will also see how things change over time.

Matej: Yeah. So you have some threshold of new players, and if you make any changes to the game or in your build, you can compare apples to apples.

Shamanth: Exactly you can do before and after pre post analysis.

Matej: Yep, exactly.

Shamanth: Yeah. So Matej,  you outline these three stages, and obviously all of them are crucial. Does this approach change if you have a game that’s fully ad monetized? Or let’s just say this is a non gaming app, say a subscription app. Does this approach change? How would you suggest changing this if at all?

Matej: I mean, it’s still monetization, so you still need to somehow improve the monetization metrics. And during the monetization stage, even if it’s an ad monetized game, so you need to find the proper placements, you need to work on the eCPMs, you need to work on the monetization, you just like different methods. 

If it’s subscription  based apps, I would say, I don’t have that much experience with soft launching subscription based apps. But again, the structure and the framework is similar. You just need to adjust it to the product you have. Again, if it’s subscription, then again, that is the monetization method. So you need to find the proper balance. You need to find the proper pricing for the subscription and so on. 

Shamanth: Sure, sure. And I imagine you can just adjust this approach, like for retention test, you could look at the free trial, the tutorial and ub monetization test, you just do the monetization subscription completion. 

Matej: I mean, yes, but you still need to look at the players. Well, not exactly like for apps – like the people funnel and people onboarding, see if there is another problem in the onboarding process. If I download the app, I still need to create the account, start like doing anything in the app. So all these flows, we need to check that as well for any app.

Shamanth: Certainly, certainly. And what do you see are some of the common mistakes that developers make during their soft launch?

Matej: Yeah, so the first thing, first of all, starting the soft launch on iOS. I don’t know why Supercell does it but I mean, it’s Supercell, so they can do whatever they want. But then, a lot of people from Finland start on iOS, which can be a slight pain in the ass, because it takes a little time to get the game approved. So I would say this would be the first thing. 

One mistake that people make is they think that soft launch can be done in one month. Because they think: we did everything needed in development. And now we just launch the game in a couple of countries and do some iterations. And then the game is ready for global launch in one month. 

But well, definitely not. I mean, you still need to be in this soft launch for three to six months to see how the players or the cohorts mature. There’s still a lot of things you need to test like creative testing, testing the new UA channels—there’s tonnes of stuff you need to do before going global. Yeah, and you just can’t do it in one month. That just can’t happen.

Shamanth: Yeah. And I think it’s also because a lot of people are like, we’ll test for one month, and everything will be amazing. But that rarely happens. And, if things go wrong, it takes you at least a couple of months to fix it. Certainly that’s understandable. What is the typical duration that you see – and you recommend doing a soft launch for?

Matej: Well, I would say at least like six months. I mean, you can do a soft launch in three months, but it’s just, it’s like very, very tough, but it is doable. It’s doable, but six months would be a good timeline. And you know, it depends on several factors like, if the game or the company has money in bank accounts or they need to launch the game as soon as possible. So there are, like, other things that you need to think about. And then like, I would say, like half a year, that’s something like a healthy time frame.

Shamanth: Sure, sure. And if they have a six month soft launch, what percentage of time goes into each of these three stages?

Matej: So basically, the technical stage is just one or two weeks, depending on the number of issues we will find. And then each stage, three months, basically, would be something that I would personally aim for.

Shamanth: Cool. That certainly makes sense. And hopefully, I think what listeners will take away from this is that it takes time to get a soft launch right. And it makes sense to put in the time and effort to iron out the details before you go global because that’s really when you can get the most bang for your buck. And I think that I hope that’s what people take away from this.

Matej: And also, if I may just jump right into this with one comment also regarding the global launching – like going global, but also

We need to speak about the other side, not only going global but also you have to sometimes kill the game. Because if you don’t see your KPIs improving, and you’re stagnating on the same numbers, then it’s just better to kill the game right away, because you are missing out on the opportunity of creating something better and something else.

So this is something that also needs to be discussed, you need to keep in listeners’ minds that, you know, you either kill or go global.

Shamanth: Yes, that’s an option just as well, that’s always an option on the table. And I think that’s also something that developers miss when they’re like, oh, I have a soft launch for one month and I will go big. And I think it’s important for them to realise a soft launch might result in you having to kill the game. It’s not pleasant to confront, but it’s certainly a possibility.

Matej: Yeah, unfortunately, this might happen and if you find out that you’re going to kill the game, the sooner the better, I would say, because we’re just wasting time on the game that just you can’t improve.

Shamanth: Right. Right. Certainly something to be very, very mindful of. Certainly something to be very cognizant of. Matej, as always, this has been incredibly instructive.  Excited to have you again. And for folks who are listening, if you guys want to go deeper, please check out Matej’s course on the Mobile Dev Memo Academy – Soft launch strategy for mobile games. 

As Matej pointed out, the very same approach is very much applicable to even if this is not a game that you’re looking to soft launch. Definitely, I think Matej goes much, much deeper into not just these things but also a lot of other aspects of soft launching – UA, creative testing, ASO etc. a lot of things. Definitely check out his course. Matej, on that note, perhaps is a good time to wrap. Thank you so much for being on the show.

Matej: Thank you very much for having me again. I had a good time. Thank you so much.

Shamanth: Thank you, sir.

A REQUEST BEFORE YOU GO

I have a very important favor to ask, which as those of you who know me know I don’t do often. If you get any pleasure or inspiration from this episode, could you PLEASE leave a review on your favorite podcasting platform – be it iTunes, Overcast, Spotify or wherever you get your podcast fix. This podcast is very much a labor of love – and each episode takes many many hours to put together. When you write a review, it will not only be a great deal of encouragement to us, but it will also support getting the word out about the Mobile User Acquisition Show.

Constructive criticism and suggestions for improvement are welcome, whether on podcasting platforms – or by email to shamanth at rocketshiphq.com. We read all reviews & I want to make this podcast better.

Thank you – and I look forward to seeing you with the next episode!