---

About Itai : LinkedIn

Check out Voyantis’ signal engineering gap analysis

FULL TRANSCRIPT BELOW

Shamanth: Welcome to the Mobile UA Show. Today’s episode is a part of our series Intelligent Artifice, a deep dive on how AI is transforming performance marketing. Since 2018 and over 250 plus episodes, we’ve been deconstructing how the best performance marketers actually win. Every week I sit down with the operator shaping it. Or we tear about the ad systems behind the highest performing advertisers in the world. I’m Shamanth Rao, founder of RocketShip HQ. Let’s get into it. My guest today is Itai Kafri, who leads product growth at Voyantis, a predictive LTV and signal engineering platform for mobile apps. Itai has worked in mobile user acquisition since before Meta introduced OCPM in 2014, back when advertisers controlled every bid, every campaign, every ad set manually. He spent many, many years reverse engineering and understanding what goes behind the scenes of the algorithms and what truly drives performance. I thought this would make for a great episode because the vast majority of advertisers look at algorithms as black boxes that cannot be controlled and cannot be impacted. But Itai shows how it is absolutely possible to impact the algorithm and to control the algorithm by controlling the signals and the events that flow into the algorithm. And for that very, very instructive read of how the algorithms work, I highly recommend this episode, and I’m excited to present to you Itai’s perspective I’m excited to welcome Itai Kafri to Intelligent Artifice. Itai, welcome to the show

Itai Kafri: Thank you. Happy to be here

Shamanth: Yeah. I’m excited to chat with you, for very many reasons. You know, we’ve overlapped for a very long time in different companies, but you’ve seen how things were since the very, very early days of mobile. And so you’ve seen a lot, and you’re doing some very interesting work with really making the best of what a lot of people think cannot be changed, which are the algorithms. Uh, and I think I find that very fascinating, and we’re gonna dig into all things signal engineering today. You know, I wanna start by talking about how the channels and algorithms have changed over time. Take us back to about 2014. What were some of the changes that Meta started to make around that time? How did running user acquisition look like before that? And what was the result of some changes that Meta made?

Itai Kafri: So great question because I think that it really leads into why signal engineering became a thing. So, back, back then, you know, before automation, you as an advertiser, were the optimizer, right? Uh, you had many, many different ad units, and you would optimize their bids, their creatives. I mean, you would bid for creative, you would bid for an audience, and you were part of the auction itself. You had total control over the inputs and, and, and how you and, and your skill would basically be, how well you optimize. Around 2014, Meta, uh, released oCPM, um, and from there on, the platform started doing the bidding on their side. So we no longer have access to the auction. We no longer, uh, do any bidding. And, and you stopped basically handling the channel’s hundreds of levers, right? Instead, you are now defining a goal, you’re defining a budget, you’re defining an outcome. The advertiser’s job flipped from controlling the bids, the inputs, to defining the outcomes. And since then, the automation just kept on going, right? Advantage Plus campaigns, PMax, UAC, Smart Campaigns, all those different, automate- automated, automated campaigns, continue in the same, in the same direction. You are no longer operating the auction, you are deciding what to tell it and how you want it to optimize for you. And that turns out to be, the whole game. So it may sound a little bit harder, but it may create the impression that you no longer have controls, but that’s not really the case, and I think that that’s part of what we’re gonna talk about today.

Shamanth: Yeah. Right. And certainly there’s been a lot of automation, and it’s only increased over the last decade or so. And in preparation for this call, we spoke, you said in spite of the automation, there is a glass ceiling to what outcome you can achieve. You know, you can’t just specify outcome and sit back. Talk to me about that. What is the ceiling? What, what are some of the things that advertisers have to do as a result of this?

Itai Kafri: I mean, we all saw huge improvements with these automations, and yet we still have a glass ceiling, as you mentioned. So the channel is now optimizing towards whatever you tell it to optimize towards. However, they, they, they put some limitations there. They can only optimize towards short-term observed outcomes. Those come in the, in the configuration of what they call a conversion window, right? When you set up the campaign. Most commonly used one is seven days, from click, and actually in Meta and in TikTok, that is the max limit. That’s the ceiling. In Google, they do have some more flexibility, but that is effectively the ceiling. So you can define the outcomes you want the channel to achieve, limited to seven days, and that bites the moment you have a trial or a subscription or a longer payoff or anything like that. It’s just not enough to tell the channel what is a good user really, within seven days from click

Shamanth: You know, even if you can get a real revenue event, not going to be always predictive of a long-term LTV. You might have somebody making a purchase. You have no idea if they’re gonna make purchases first purchases for the next one week or one month or two years,

Itai Kafri: Yeah.

Shamanth: And how much they’re gonna pay.

Itai Kafri: It, it’s the easiest to see it on a subscription, right? A, a user subscribes. Does that mean you will retain or, or not? You, you don’t know. The channel has no way of indicating or, or knowing that, so they optimize towards a subscribed, retained user. They can only optimize towards that event called subscription

Shamanth: Yeah. And one place where this delta, if you will, manifests is how marketers value users versus how finance teams and CFOs value users.

Itai Kafri: Yeah

Shamanth: Again, I worked in very many companies, including public companies, where I’ve seen this gap, and I’ve also just basically seen the finance teams use completely different reporting as compared to marketing teams. Talk to me about why this gap exists and why it’s important a marketer or a financial person or an exec or a founder. Yeah

Itai Kafri: It’s important for everyone because you want the organization to work around the same objective, but, but why does it exist? So, both sides are doing their job very well, right? They’re just measured on, on different outcomes. Growth marketers are measured on short-term measurement, things like, you know, CAC, volume, D7 ROAS, D30 ROAS. And it’s because the tool set that the channel has to provide is a short-term tool set, right? You can only optimize towards D7 revenue or D30 revenue or, or that, you know, CAC. And so you can’t expect a growth marketer, to be measured on something they don’t see six months, into the future. They, they, they can’t take any action next week to improve that, or to see that. So if you put a CFO and a growth marketer in the same room and ask them, you know, “Define a great acquisition, define a great user,” the growth marketer will probably say something like, “Give me a user with a low CAC and strong D30 ROAS.” The CFO will say, “Give me a, a loyal customer who stays loyal and is still ordering six months into the future.” Both answers are rational, both are good. One of the reasons that I think signal engineering is so interesting is because part of signal engineering, in the way that we perceive it at least, is the prediction. And once you have a prediction in place, you basically can, can bridge that gap between the two languages and align everyone around the same objective and the same goal

Shamanth: Yeah. And you also said when we were preparing for this, right? So the real reason you need the prediction is because the deterministic event is not enough. And, uh, by definition, the prediction is not precise, and therefore, you need to have a confidence level, and that’s a trade-off. So talk to me about the trade-off itself, and how you suggest marketers deal with this, and maybe you can also provide examples for folks that may be interested. Yeah

Itai Kafri: This is gonna be a bit of a longer answer. So, Most Teams thinks that a prediction, you know, a prediction is all you need, and once you have a prediction, you’re ready to go. Now the model is very important and the prediction is very important, but it’s, very different to work with it on a media channel, when you compare it to a deterministic event. Confidence is one of those things that makes a prediction different. A deterministic event is very clear, right? Someone went into the website, purchased a TV. You have an event name, an event ID, you have a timestamp, you have a, a value. You know exactly what happened, and you fire it the moment it happens as close as possible to real time. And that makes it easy. But once you, look at a prediction, that additional attribute called confidence, creates some trade-offs. It creates a timing trade-off. If you wait for the confidence to go high, you may be a little bit too late, and you missed the, the window where the auction is really hot, where you can make stronger impact. If you send it a bit too early, then the confidence is low and you’re basically feeding the channel with bad data. And we’ve tested this. We’ve tested this thoroughly. So we ran experiments where one group, uh, the control group, received the signal within hours, and then the test group, we started adding artificial, artificial latency into it to see what happens when you delay that signal further and further. Assuming it takes you more time to, you know, to predict or, or to get the confidence, what happens? And the further you push that signal, the lower the performance. There are some significant drops, uh, when you pass the 24-hour from click time, you basically move from a D1 conversion window to a D7 conversion window. That’s a big trade-off. But the further you push it, the worse the performance. So you do want to get that prediction sent to the channel as early as you possibly can. And that’s, that’s that confidence thing. Now, what we found when we talk to clients that try this and, and build their own prediction models is that they come up with a, static kind of, you know, definition. They’ll wait three days, they’ll wait seven days, depending on, you know, the prediction model and, and they’ll send it the same for all users. That’s kind of like, you know, if we, if we compare that to bidding, that’s kind of like bidding the same on all ad groups or on all campaigns.

Shamanth: Mm-hmm.

Itai Kafri: When you want to optimize, you need to be granular. And if you can send the prediction earlier for one user versus another because they have a different journey and they mature differently and the confidence matures differently, then you should do that because that’s really optimization and you have the opportunity to optimize user level, which is something we never had before. We were bidding on an ad group level, on a campaign level. You never had the opportunity to actually optimize the user level and now you have the opportunity, so don’t miss on it, right? And, the way that we do this is we calculate, uh, the prediction for every user every hour, up to every hour, and then it gives us the flexibility to decide which of these pass a certain, um, confidence threshold and can be sent to the channel. That allows us to send very early signals for users who do mature, but for those who don’t, we wait a little bit longer, so it gives us that flexibility. That’s how we solve it for these

Shamanth: Right. So every prediction has a confidence level. And from what you’re saying, look, in an ideal world, you could basically wait 365 days and get an extremely accurate signal, but that’s not gonna help you run your acquisition, if I understand what you’re saying. What are some of the elements or variables on which the confidence does depend on? You said, you know, you’re running these predictions every hour.

Itai Kafri: Hmm

Shamanth: What goes into making a prediction every hour that is better than bidding the same on all ad groups, basically?

Itai Kafri: You know, before I give that answer, I just wanna, um, say one more thing about that confidence thing because the confidence is just one dimension. Um,

Shamanth: Right

Itai Kafri: when we think about optimizing user level signals, we also think about the value range, managing outliers, late bloomers that you don’t even see at the beginning. They will bloom way out there. So all of these are different trade-offs that are relevant for a prediction model and don’t exist in deterministic. And the way that I see signal engineering is really balancing all those trade-offs. That’s the optimization layer. Now to your question, um, what goes into the prediction? The more data, the better. Prediction, a lot of, you know, I’ve seen clients try to build that prediction based on events that they see in their Appsflyer or their, you know, their MMP, whatever that would be. That’s, um, not really enough. So I, I wanna give an example, an intuitive example for a moment. Let’s say that, you want to predict, which marathon runner is gonna win a marathon. You look historically at all the recorded marathons that you can find, and you see how much time did it take every runner to finish that first lap and the second lap and the third lap, and then you compare that to how they finished the race in total. And now comes a new marathon, new runners. You never met them. You never seen them. You don’t have any historical data on them. They start, and you basically use those lap numbers to predict by lap two or by lap three how well they’re gonna finish that race. And, it may give you some indication, but let’s take that same story and add some more data points to it. Let’s say that I’m running the same analysis on thousands of runners historically, but I’m also looking at, um, uh, things like how often do they drink? When do they take that first energy bar? Um, how high do their feet rise above the ground when they run? Um, how their hands movement, you know, kind of as they run. And I put all that together. Every individual one of these doesn’t sound interesting, but when you put them all together, it gives a much more accurate prediction. So now in our world and, and, and going back to, what we call, building model, it would be what does the user do and how do they behave? Um, say, a game. There’s an event called pick, you know, avatar select. I wanna know how many swipes they did before they selected that avatar. I don’t want to see the events that they did. I wanna see how they behave. So we’re looking at their users first-party data like OS, like, uh, their geo, their device type, but then we’re looking at what did they answer in the onboarding quiz or onboarding questionnaire? How much time did it take them to answer those questions? We’re looking at, which clicks and sessions and what did they view and which, product features did they use? Um, and obviously the transaction count as well. So we’re looking at all that to build a good prediction model. That’s what goes into it. Now, if you take that and say, “Why do we predict every hour?” Because users engage Or don’t engage with is also a good signal. Um, and so we know what they do, and we track it down, and I, and I said up to every hour because if, you know, if really in the data there’s built-in latency and it only updates every three hours or every five hours, then there’s no reason to predict every hour. But we try to predict as high cadence as we can in order to try to get, capture that confidence as early as we can

Shamanth: Yeah. Right. So from what you’re saying, the more variables you have, the better your predictions will be. In the marathon example, it would be, what energy bars they’re eating. What did they drink last night? How much did they eat? What is their normal diet? You have a much richer picture. What would you say to someone that says, “Look, uh, non-purchase events are not really very good predictors of purchase behavior.” what have you seen in your data, and what would you say to somebody that says something like that?

Itai Kafri: Um, we, we don’t ask that question. We ask for as much info data that we can and put it into a machine learning model. Um,

Shamanth: Appsflyer

Itai Kafri: we find that there is a strong correlation between events that sometimes you don’t every time we have a client, I’ll just, I’ll just talk about this and from, from my perspective, right? After we finish building the model, we test it, we see how accurate it is. We then display that to the client. We show them we have a model evaluation kind of meeting, and we show them everything, and we show them what are the, some of the main features that went into the model that were heavy or important in the model. Out of ten features, nine of them, the client would say, “You know, I knew, I knew that’s important. I knew that feature would be, or that parameter would be important for the prediction.” There’s always that one that people scratch their heads and say, “How is that related to the user’s LTV or to the user’s retention or to their propensity to buy?” And there’s always that one. So, it’s about the, the combination between all of them together that makes it a strong model

Shamanth: Right. So it’s not a variable in isolation, it’s the interrelationship between those, right? So it could be, you know and this, if I understand correctly, is the reason why if you try, try, you know I’ve seen this, and you can tell me if this is incorrect. What I have seen is a lot of marketers and analysts, I’ve seen them say, I know who complete levels five are likely to purchase because I see a strong correlation between level five and purchase.” then they optimize for level five.

Itai Kafri: Yes

Shamanth: see purchases. And yeah, go ahead. You, you wanna speak to this dynamic? Yeah.

Itai Kafri: Yeah. It, it’s a very big cohort, five, it, it’s another proxy. So every time you pick a proxy to optimize towards, you’re, you’re giving up, um… The, the size of that cohort has variance within it. So out of those who reach level five, some will make one purchase, some will make… Maybe some will make zero, but, but a lot of them will make, you know, a few, and a lot of them will, will retain and make a lot. And so the, the, the bigger the cohort, the more variance there is in it. The more granular

Shamanth: Yeah

Itai Kafri: your, um, feature, we call them features, right? What goes into the model is features. The more granular your feature matrix or feature library is, the more accurate you can be with that prediction

Shamanth: Yeah. Do you need a critical mass of users/purchasers to be able to come up with something reliable?

Itai Kafri: I’m gonna give you two answers for this, two answers for this. Um, there is one answer, which is how much data you need to build a prediction, and there’s another answer, how much volume you need to test the impact on the media, right? When you start running this campaign. So I’ll start with the first one. The first one, we categorize data into categories. We say first, you know, user is first-party data, anything the user declares on themselves, so that would be the onboarding quizzes, the onboarding questionnaires, that declarative data, product engagement data. Everything they engage with the product typically comes from a product analytics platform or something like that. And then transactional data, and at the very end, the last category would be, um, attribution, performance, you know, which creative adword um, sorry, keyword, and so on, brought this user. And if there’s any incrementality test that you have, all that goes into performance and, and attribution. When you have all these categories, even if some of them are thinner, some of them are, richer, um, if you have all these categories, you’re likely going to be able to build a good model. The less you have, the less accurate the model will be. Um, and I’m gonna talk about accuracy in a moment because that’s a very important, uh, uh, piece to discuss. But, all that goes into building a good model. Building a good model is just one step. Optimizing it into a signal or translating it, engineering it into a signaling strategy is a whole different story. And when you want to start signaling to the channel to drive uplift, then you need to ask that second question: Do I have enough volume of events? Because this is translating from a bunch, you know, thousands of features to build a model into a single event per user, and now I need to train the channel to drive better users with that single event. And that’s where things become, you know, volume becomes a, a bigger issue from my experience, where if the client doesn’t spend enough money compared to the CAC and the, the, their signal density is low, it becomes very difficult to pass learning phases and to test it out. So that’s, that’s, you know, on the volume side. So you ask me how much data do I need? You need data to build a model. You need data to run it, or volume to run it.

Shamanth: Yeah. Yeah. And when you talk about, you know, not just finding the event, but passing it back to the algorithm, I think that’s a very critical piece because otherwise you just have a bunch of insights. You’re not really doing anything with it. Can you speak to the recommended approach for passing the event back in that, what is more sophisticated in terms of an approach compared to the level four that we just talked about?

Itai Kafri: So I’m gonna combine my answer here because I wanted to talk about accuracy, but this actually, uh, puts things together. What does a channel need? What is the channel looking for? So first off, a level four, event is what we call a binary optimization. You’re, you’re signaling to the channels, just one signal. It’s a one or a zero, right? The user did or did not. And you’re ranking all your users randomly by doing so because everyone who did pass it, they’re all equally valued by the channel. If the user reached level four, that’s a good user. The channel doesn’t care about anything else, right? Um, we are optimizing in a way that we want to use tROAS or value-based bidding or VO. We wanna use a value-based bidding, and then we want a variety or wide range of values, In the same way that you would use, you know, VO for d7 ROAS, right? Um, and so once you do that, um, you start thinking of what does the channel need to see in that range. And the reason I said this is also connected to accuracy, at least in my mind, is because most of the people that I talk to that are not really into the weeds, when they think of a PLTV media strategy, they think that their LTV model needs to be accurate, accurate to the dollar, right? Or by measuring dollars, right? And that’s a bit of a misconception. The channel doesn’t need to see what each user is worth. The easiest example would be if you have only two different packages in your app. One package is $47 worth, and the other is $54 worth. And I’m making it as simple as I can here. The difference between them is about 15 percent, give or take. You can send those values to the channel as they are – And you’d be super accurate if you do that. But the channel doesn’t differentiate between them very well because they’re very close together. If you are thinking about what you’re sending to the channel as a signal, then why not send this one as $10 and this one as $50 or $70? Now you’re creating differentiation for the channel. So what the channel is looking for is accuracy in dollar amounts. The channel is looking for ranking, for separation to make sure that it’s learn, you know, the learnability is, is there. And when you think about it from that angle, then the entire signaling strategy is built around that. So when you have, a certain value range, if you have an outlier, say that the value range, the typical value range is anywhere between $50 to a $150, that’s most of your users. Comes this outlier worth $5,000. RMG, you see that a lot, right? This user comes in worth If you start sending those, and they’re only two percent of your population, if you start sending those, it may devalue everyone else. So you’re devaluing your whales for your outliers, and you won’t get those outliers because they’re so scarce. The channel will try bidding high for users, but they don’t come in,

Shamanth: Yeah

Itai Kafri: and then you’re seeing increase in CPAs, but you’re not seeing it backing out. So a, a signaling strategy would say, “Okay, that’s the prediction. It’s the user is worth $5,000.” But when we send it as a signal, we can cap it at $170. If the value range is a 150, give or take, $170 is already putting that user at the top quantile, at the top percentile.

Shamanth: Right, right

Itai Kafri: So signaling strategy is not about here’s the LTVs, let’s send them. It’s about how do we actually manage this? And by the way, it’s a moving target. It’s an ongoing, um, uh, reinforcement learning between you and the channel. If,

Shamanth: Yeah

Itai Kafri: you start doing this, the user mix changes a little bit. You wanna react to the user mix change. If you increase your budget and now you have way more signals, you can create more differentiation by just deciding that the low-value predicted users, let’s not send them at all. So suddenly you can start capping the bottom end. If the budget goes down, you don’t wanna cap them because you’re gonna be in a signal density issue. So that is a moving target that you play around with, and it’s exactly the same as we used to do on bidding back in the day when you had automated bidding by all these third-party tools. We’re just taking that upstream and instead of managing bids on top of the auction, which we don’t have access to now, we’re sending the signals, and we’re optimizing them constantly by seeing what’s going on with the user that’s coming back, the user mix that’s coming back, and the performance that’s coming back

Shamanth: Yeah. No, that’s interesting. What you’re saying is you’re not just looking at the user event stream to understand who’s the most valuable user, you’re also looking at the cost you’re paying the algorithm because you might identify the most valuable users, if the algorithm is going to charge you 10x, it’s just not gonna be worth it. So part of the signaling strategy, if I understand correctly, is managing the algorithm’s costs through reinforcement learnings. Did I

Itai Kafri: algorithms cost

Shamanth: that correctly?

Itai Kafri: And the user mix. Users are different, and the moment you change your optimization, you’re getting a different mix back. So, I’ll

Shamanth: Yeah.

Itai Kafri: A real example. I’ll give a real example here. We had a client that built their own model. They were extremely happy with their model. It was very accurate based on measurement. Typically, measurement uses MAPE. They used MAPE to measure the accuracy. They were very happy. They turned it on as the first, PLTV campaign. Performance tanked within two days. And after a bit of investigation, the reason it tanked is because their model, every model will be inaccurate here and there. It, it’s a given. Um, there was a very small population that the model didn’t recognize as an important, significant population, so it didn’t differentiate them, and it, created the bid across, you know, the, the value across all users together. That population was Google Search Partners population. Very small population on their, on their end. But when you overvalue them and send that as a signal, the channel then sees that and says, “Oh, there’s a pocket of opportunity.” These are users that are cheap to buy, and they’re valued high. That high value is a mistake. It’s not real, right? Because the value the population was so small that you can’t recognize it. But now the channel is starting to send more users from Google Search Partners. And for that client, that happened within two days. Up to 45% of their population became Google Search Partners. You need to so the user mix is changing, and as you do this, as you change the user mix, you need to see what’s coming back and adjust your signal back and forth. And now, okay, so now I have a bigger population here. Can I add a multiplier to decrease it, or can I do this, or can I do that? Or no, do I need to actually go back and retrain the model? Right? So that’s a reinforcement learning system

Shamanth: Right. So if I hear right, the model may be accurate at one point of time, but if the model’s results start to drift, you need to adjust the model.

Itai Kafri: Exactly.

Shamanth: Correct me if

Itai Kafri: Exactly. And so we separate that, right? Signal engineering. Signal engineering will take the immediate actions right away because you can’t train the model

Shamanth: Yeah.

Itai Kafri: Everyday. And when the drift is, is significant, we retrain the model, but we have a immediate reaction because the auction is very dynamic

Shamanth: I think that makes a lot of sense. Since you spoke about outliers and whales, and I like and appreciate that example because it talks about how algorithms can inflate costs for whales. With that said, if you are a product that is dependent on whales, you talked about RMG, there’s social casino, I’ve certainly seen the monetization curves of a number of these products. What, if anything, can signal engineering methodology do to help you acquire more whales? Or is that not even the, the right question to ask?

Itai Kafri: No, it is. Um, so let’s differentiate for a moment. It’s not a term that’s used a lot, but I, I, I think it, it needs to be differentiated. Whales, are good. They’re part of your business. You want to grow your whales.

Shamanth: Yeah.

Itai Kafri: Outliers are noise. And so when we look at the entire data set, we say, “Wait, let’s, let’s divide it.” These are the normal users. Uh, these are the whales. Whales should be around, I don’t know, 10%, 5%, 20% of your user base. They can’t be a one percentile. If that’s a one percentile, it’s, it’s an outlier. So the outliers are noise, and so y- there’s a statistical different statistical model to differentiate them. What you wanna make sure is that you’re, um, building the model in a way, and, and the signal engineering, the whole process in a way that would, um, um, not cause the channel to chase ghosts and those outliers that are extremely high value. We’ve seen th– I, I’ve seen this, you know, throughout my career a million times where outliers can cause the campaign to go haywire. Um, um, and, and you wanna when you’re controlling the signals in a, in a good way, you wanna control it smart. And, and if you look at… If you do– just do D7, then the differentiation is not that big. But if you start looking at LTV D180, it becomes so big that if you that you have to start capping the outliers out. So you’re not, you’re not filtering them out. You’re– They’re part of the population. You’re just capping their value so it doesn’t cause the campaign to go crazy, and you’re basically bulking them with your whale population

Shamanth: If I had to take the flip side of that and say maybe there’s a world where I have a hyper whale I’m prepared to spend $1,000 CPI for that, will not the value of the whale be counterproductive in that situation?

Itai Kafri: It’s a probability game. So

Shamanth: Right

Itai Kafri: think, think of what the channel actually controls. The channel controls impressions. That’s all they know, right?

Shamanth: Right

Itai Kafri: Now they’re trying to predict which of these impressions are going to fall into the hands of a user that you’re trying to acquire. Um, you know, work, uh, work backwards from this, um, outlier, you know, you call them hyper whale. Look, work backwards. That is going to be a 1% of the payers. What is the percentage of payers out of the installers? What is the percentage of installers out of clicks and out- clicks out of impressions? That means that they are trying to chase something that is so scarce. Now, what ma– And, And that’s what causes these campaigns to go haywire really, right? They’re trying to put impressions in front of people they think may be that huge outlier. They don’t end up to be the outlier, but they’re paying a lot for that impression. They’re, they’re bidding high for those impressions. So you end up paying more for all of your users, and you’re not getting more, you’re not getting more outliers. You know, you’re not getting more of

Shamanth: Yeah. Yeah. Yeah. Yeah. Yeah. I, I get what you’re saying, right? So because the very act of bidding high for these outliers is going to the algorithm, it will signal that you are ready to bid more for everyone, not just this one user

Itai Kafri: And it’s very hard to find them. It’s very difficult to find them with the data that they have

Shamanth: Yeah. Yeah. You know, this, uh, it, uh, this makes me think of RTB- Real Time Bidding, where, again, you can tell me how different the methodologies you’re describing are compared to how RTB works. It’s my understanding that with RTB, it’s also a very probabilistic thing where you say, “Hey, here’s the probability that this user is gonna be extremely high value. They’re gonna bid X amount.” So how would you compare signal engineering with, with an RTB system does? Yeah.

Itai Kafri: I would say that it’s tenfolds more difficult in signal engineering and for two main reasons. The first one, it’s not only looking at the users and, and bidding, you know, for, uh, for those cohorts that you wanna bid for. It’s, um, it involves a prediction as well, so that’s another data science piece that goes in there, uh, which is not the most difficult one. The most difficult one is that RTB is bidding on ads or bidding on ad sets, ad groups, or campaigns. Um, These are big, you know, they, they contain a lot of user, You’re not doing it on a user level. So n- take, take build a prediction, and now you’re signaling on a user level. I see that as an amazing opportunity because if you were able to bid… I mean, I remember when Meta, y- y- we, we had the opportunity to bid on, on ads, then they moved the bidding to the ad group. But through the API, they still allowed bidding on the ad level for a while, for like a few years. And during those few years, we enjoyed that because I, I worked in a third-party company that did bid, bid automation and, and we actually were able to bid on the ad level. The more granular you’re able to do this, the better you are. Um, So here

Shamanth: Yeah

Itai Kafri: engineering is on a user level. That, that’s a, that’s a game changer in terms of volume of, of calculations and what needs to be done in the back end, right?

Shamanth: Yeah

Itai Kafri: So it’s a bit more complicated, but, but it, but the logic is the same. The, the i- The concept is the same concept

Shamanth: Sure. Sure. It’s conceptually similar, and I would also point out that even though you are pre– making predictions at the user level, you don’t have user-level data at the al– with the algorithms for various privacy reasons. So, for the alg– at, at the algorithms level, you’re still looking at aggregate data, and you’re trying to correlate that with user-level data. Did I understand that correctly?

Itai Kafri: Um, We connect to our clients’ data lake, so we see their product engagement data as they see it,

Shamanth: Right

Itai Kafri: we do it with no PII. So, uh, we do keep it privacy, you know, everything. Um,

Shamanth: Yeah

Itai Kafri: we have all the certificates, and we work with fintech, which are very difficult. But the way that we do this is because we don’t have any cross-client learnings, so we learn only one client at a time.

Shamanth: Right

Itai Kafri: we do– you do need that user-level data, uh, to be, to be accurate. Um, You can mask it. You can play around with it. So, for example, if I say a payment type is very important for me, I wanna know if the user paid with Visa, Mastercard, or Stripe. That can define that, that’s part of the feature matrix that goes into the prediction. I don’t need to know if it’s Visa, Mastercard, or, or Stripe. You can call it A, B, and C. I just need consistency across all your users to see this guy paid with A, let’s see what it looked like at the end. This guy paid with B, let’s look at what it looked like at the end. So as long as everything is consistent, um, it can be masked, but we do need user-level data. Yeah

Shamanth: Of course. Yeah. Yeah, of course. But I think what I was getting at was, yes, you do need user-level data to, to do the prediction, but

Itai Kafri: what

Shamanth: happens behind the algorithm is gonna be aggregate data, not user-level. So you have

Itai Kafri: What we send to the channel

Shamanth: What you send to that channel is probably gonna be user-level, but what the channel does with it and what you’d notice from the inputs you receive from the algorithm is going to be aggregate They’re not gonna be… You’d say, “Hey, this is what happened for this ad group or campaign,” but you’re never gonna get a user-level data from the algorithm

Itai Kafri: Oh, but we see… Oh, but we do… Oh, okay. So I get it. We get back from the channel, you know, performance data in aggregate like everyone else. But in the database, we see individual users, and we see the changes in the user mix. So then we can see which users do what and how, how did it impact the, the overall user mix. And so we do see that, and then we can identify, you know, that’s how we identify

Shamanth: Yeah

Itai Kafri: those new biases that u- that new, that new category of users that suddenly became bigger, like the Google Search Partners. Suddenly we recognize more of those, and we need to address it, and we need to, uh, react to it

Shamanth: And that’s not deterministic, but again, the stronger your predictions are about what the algorithm is sending, the better you can influence the algorithm.

Itai Kafri: Exactly.

Shamanth: I understand

Itai Kafri: Yes

Shamanth: Yeah. Yeah. Okay. Uh, that’s– Yeah, that’s so fascinating. That’s so interesting. Also, just because a lot of what happens behind the algorithms is a black box for the vast majority of people. And I think what I like about everything you’re talking about, Itai, is that you’re not saying it’s not a black box. It is a black box, but

Itai Kafri: They’re patterns

Shamanth: Black Box has some predictable probabilities. uh, The approach you’re describing is like, how can we make this less of a black box? It doesn’t have to be a hundred percent, but it doesn’t have, you know, or how, how do you make this less of a black box?

Itai Kafri: Uh, that’s an interesting observation that you just said, and I-I-I’ll, I’ll add onto it. Um,

Shamanth: Thanks

Itai Kafri: predicting user value, we predict across the entire user base that the client has, and we can predict for every user, regardless of which channel they came from. Um, The channel will be just another feature that we use for the prediction. But signal engineering on our side is channel by channel, and we have to unpack or reverse engineer as much as we can that black box. We only support three channels because of that, because every time we want to add a channel, we need to run thousands of tests just for the sake of the example that, you know, test that we did to see what’s the impact of a latency. Um, we also need to do that on what’s the impact of accuracy. How do we trade off different channels behave differently? In Google, you can restate the value. In Meta, you can only increase the value that you sent. So if you sent a prediction in Meta, you can’t decrease what you already sent. Uh, in Google, you can. And that makes the signal engineering very channel specific and very use case specific.

Shamanth: Right.

Itai Kafri: And yes, they are black boxes. With that said, there are patterns, and the more you invest in it, the more you identify them. And really making this whole thing work is the intersection between the data science and the art of, of finding all those things and working

Shamanth: Sure, sure, You know, which also makes me wonder, right? So Does– I, I’m– I would imagine that the algorithm’s behaving a certain way, the users are behaving a certain way, and just say the creative makes changes or there is an external factor like the World Cup or Christmas that has nothing to do with either the users or the algorithm per se. How, if at all, would the signal engineering approach account for this?

Itai Kafri: Well, there are some things that signal engineering will hit directly. If suddenly because of these things you have a lower volume of signals because you cut your budgets because of, I don’t know, Black Friday, You’re gaming, you cut your budgets, right? Um, or the other way around. Black Friday, you’re e-com, you increase your budget. Those things will immediately impact how signal engineering works because now it can– it has more wiggle room with a number of, uh, um, volume of signals if you increase your budget, and it can, it can create more differentiation, cap it more here, cap it more there, do that or do that. Um, But there are things that are just generic and, and if, if the entire activity changes this way or that way, you measure and you see if it happens to everything, including what runs through the PLTV signal and what doesn’t run through PLTV signal. If it’s all the same, then it’s part of what the auction is, is doing and, and you can’t always impact that, and we can’t always promise that it will be, you know, it’s, it’s still an optimization challenge

Shamanth: Yeah. Yeah, yeah, yeah. For sure. Yeah, and you know, I would also imagine this is way signal engineering can apply is very different for different verticals. And when we were preparing for this call, you did say it can be rather tricky for gaming. Can you talk to me about why that is and what are the specific dynamics around gaming that lead to this?

Itai Kafri: Yeah. Um, and I’ll say casual and hyper casual specifically are a bit tricky here not RMG. Um, Correct. I think the, the real lesson is not that it’s tricky, but why it’s tricky because that will answer, um, for every vertical how to look at their data to identify if they should even invest in, in PLTV. So before you build a model, before you think of signal engineering, put signal engineering aside, signal engineering as the activation part of it. Before you even go there, should you invest in that direction, um, is a big question. And the whole premise of using a predictive media strategy is to re-rank the users compared to what you’re doing today. If you’re not able to re-rank your users, you’re not making a difference. So I wanna give a few examples, right? If you are running a TC- a TCPA campaign or a CPA campaign, you’re not ranking your users, you’re just splitting them into two groups, good and bad. But if you’re running a, a D7 TROAS campaign or a D7VO campaign, you’re ranking the users. The signal that you’re sending to the channel is ranked by their D7 revenue And now the question comes, if I rank the users based on how they perform at their end of their lifetime, say one year or six months or whatever you wanna call LTV in your case, um, would it make a difference? And there are a few ways, or let’s say there are three big questions that you need to answer to, to find out if it’s worth it or not. And for all the listeners on the, on the line here, this is where you wanna take notes, um, these three big questions. If you’re not there yet, if you’re not running any kind of PLTV, these are the three big questions you wanna answer before you go there. The first one is, what is the pool size? And I’ll get back to gaming in a second specifically. What is the pool size of users that you can re-rank? An example would be if 95% of your users churn by D7, then the D7 signal is strong enough because the pool size that’s left after D7 is just too small for you to make an impact. Um, The second one, even if that pool size is big, 75% of your users, for example, retain P- beyond D7, meaning for those users, maybe their value will be different at day 7 and day 180. For those, you wanna see what’s the value that happens post D7. If everyone is, is, um, um, uh, you know, front loaders, like they spend a lot of money at the beginning and then it’s small little purchases afterwards, even though you have a big pool, changing or re-ranking won’t make a big difference in your, in your LTV and your ROI at the end of the year. And the third one, so let’s say now you have a big pool and there’s a lot of revenue to re-rank in that pool, how confident are you that you can make that difference? And if I look now specifically at gaming, casual and hyper casual, what we found is that the correlation between their D7 value and their LTV is very high correlation. In other words, a user that spent a lot in D7 is gonna spend a lot in D180. And what that means is even though there’s the big pool there, predicting that value will not really re-rank them. And as we said earlier, what the channel cares about is not the value that you’re sending, it’s the, the rank. If the rank is the same, you just scaled up all the values, they’re not gonna change the delivery. The delivery is gonna be the same, so they’re gonna still bid the same for a user. That’s what happens with, uh, with, uh, many times with gaming. I’m not saying it’s a, it’s a, it’s a, you know, it’s always the case. But in many cases in gaming, we see that because it’s hyper small, hyper transactions, um, there’s just that high correlation between what they do by D7 and how they behave after D7

Shamanth: Yeah, and correct me if I’m wrong, but that would mean that the D7 metric is a very good predictor of long-term value, and that is really the…

Itai Kafri: Exactly

Shamanth: That’s all the signal engineering they need,

Itai Kafri: yeah, exactly. So

Shamanth: if anything.

Itai Kafri: if, you take that correlation, uh, the cor- a, a good correlation analysis, check what is their D7 revenue and how it correlates to their post D7 revenue, um, and say day 180, I don’t know, take those. Check that if the correlation is high, then you are already sending a good high-quality signal to the channel.

Shamanth: Yeah

Itai Kafri: think you need to invest in a, you know, predictive strategy in your media plan. You still may wanna build a predictive engine just to see what your, you know, to forecast for your stockholders, for your whatever, but, but not as as a signaling strategy, it will probably not move the needle

Shamanth: Mm-hmm. sure. For sure. Sure. Itai, uh, this has been very wide-ranging and very instructive. Again, as I said earlier, just because I think a lot of people assume, myself included, assume algorithms are black boxes, and I think there’s a fascinating look at how they don’t have to be, know. And you can absolutely influence a black box algorithm. Uh, know, and I think as you also said, you know, it, probability. Just the moment you just say, “Hey, how can I focus on the probability of something happening?” just a lot that you can unlock. This has been incredibly instructive, and this is perhaps a good place for us to wrap up. But before we do that, could you tell folks how they can find out more about you and everything you do?

Itai Kafri: Yeah. So first of all, I, uh, um, I lead our, our product growth at Voyantis and, uh, uh, the company is called Voyantis and, and you should check out our website. Um, more so, um, The three, um, big, uh, questions that I talked about, you know, the pool of users, the, the value and, and the confidence and how big of an impact can a predictive strategy do is something we have an automated tools, that can help, um, um, identify without any– for your audience, we’ll do it without any, um, commitment in any, in any way. Um, it just needs, uh, uh, eligibility in terms of, uh, being a, being a, a big enough client for us to be able to run a test properly. Um, But we can do that for any one of your clients, and, and we’ll send, you know, we’ll, we’ll, we’ll attach a link to the, to the podcast so anyone can see that landing page and, and go into that page and get a free assessment of how much of an impact can a predictive strategy do for their, um, for their user base. So I think that kind of wraps it up and, um, and you can go onto our website and, and look for, for case studies and, and explanations.

Shamanth: Yeah, and we’ll link to all of that in the show notes. Uh, this again is a good place for us to wrap. Thank you so much for your time today

Itai Kafri: Thank you