📐 How incrementality helps measure the true impact of marketing spend — with Maor Sadra, CEO & co-founder at Incrmntal 📈

Our guest today is Maor Sadra, co-founder & CEO at Incrmntal. In today’s episode, we dive into how last touch attribution has always been fraught with inaccuracies and how it measuring incrementality might just be the best way to understand what the true impact of your digital marketing is.

KEY HIGHLIGHTS

🥴 Why last touch attribution has always been problematic

🧐 What led to Uber being susceptible to fraud.

😤 How there’s a thin line between frauds and self-attribution, when it comes to claiming credit for click and view through installs

😯 How last touch attribution enabled fraud

🤦‍♂️ You don’t get direct or reliable access to impression or click level data from Facebook, or Google for mobile apps

😍 If you own your domain, you’ll get individual data about users and their impressions

🤨 How MTA can be challenging to make work on mobile

🙄 Marketing measurement is not deterministic

👍 Often if an advertiser pauses marketing temporarily, they start to see that they might be spending too much.

✌️ With sporadic incrementality testing, one will not be able to optimize marketing on an ongoing basis

🤔 Causal inference is critical while measuring incrementality.

🥂 The last touch attribution is a good way to understand creative performance..

🤵 Is measuring incrementality essential at small scale budgets?

KEY QUOTES

There is no one way to think about attribution

If you own uber.com, you would know the traffic is coming to uber.com, you can analyze it, you can come up with your own attribution model based on your user behavior. On mobile, it wasn’t the case. When the app stores were launched, there was no real way to get any tracking done.

Why last-touch attribution is flawed

And to give a hundred percent credit to whoever just tapped the user last, especially in our industry, it actually created an ocean of fraud. Because if you’re a fraudulent publisher, and you know that you just need to tap the user before there’s an install, and sometimes that’s five to ten bucks for a single install, it becomes a very easy system to actually manipulate. And it became quite messy.

How user overlap can stymie multi-touch

Because you have these behemoths that are self-reporting, the amount of overlap you’re gonna see is close to 100%. As long as you have a product that is mass-market, everybody’s gonna touch the same users. So, no one talks about the reach anymore, but Google and Facebook, they have the same reach. Amazon has a pretty significant reach as well. And so, when all of them are basically claiming, well, we show the user an ad, we generated the install, you have a little bit of a challenge to actually try to even apply multi-touch.

The reality of marketing

Marketing is not deterministic. I wish it was, but it’s not. And often, we want to tell ourselves that it’s very deterministic. We know exactly what generated or caused this install. But it’s not causality that we often know. We know correlation.

Use incrementality testing to optimize spend

And if you think offline, Coca Cola and Nike – great examples. They run a campaign for a set limited period of time, for two weeks, four weeks, eight weeks. Then they stop the campaign and rely on panel reporting, survey reporting, and sales reports. They rely on reports in retrospect to help them reevaluate what was my media mix, did I actually see an increase in sales for the product that was advertising?

Move past the need for determinism

And if they don’t, they can actually now do it better for the next time, and they do get better and better and better. But none of them tries to conclude that what they’re doing is deterministic.

FULL TRANSCRIPT BELOW

Shamanth: I’m very excited to welcome Maor Sadra to the mobile user acquisition Show. Maor, welcome to the show.

Maor: Thank you, Shamanth. And thank you for making the extra effort to try to pronounce my name correctly. It’s a common joke that my name gets butchered. I’m fine with it. But thank you for making the extra effort and inviting me.

Shamanth: Absolutely, Maor. So, yeah, definitely thrilled to have you because certainly, I’ve learned quite a lot from what you’ve shared on public forums and communities just over the last couple of months, as we’ve all tried to make sense of the changes coming our way. So definitely, I think you’ve offered some very unique and interesting perspectives on how to approach measurement in a few months, and it’s definitely something I’m thrilled to dive into with you today.

To start off Maor, until now, the last touch has been the most widely prevalent way to attribute and measure performance on mobile. And you have expressed that it’s always been problematic. Can you tell us what has been some of the key problems associated with last-touch attribution?

Maor: I think we need to remember that last touch attribution is prevalent because there were no other real options. Basically, it’s like if you come from offline advertising or desktop advertising, you know about the option of a media mix modeling, you know about the option of multi-touch attribution, you know how to actually look at the entire funnel because you do own your domain.

Let’s take Uber.

And back then, in 2012, the attribution companies were launched, all of them used UDID device tracking, later on, UDID got deprecated and replaced with IDFA and Google AID.

But still, the way that the attribution or MMP companies – they were tracking everything – impressions, clicks, wherever they could. And I think a lot because the big platforms, like Facebook, started on mobile, from day one they were self-attributing. And from day one, in order for an MMP to join the MMP program and to basically send their posts back to Facebook and so on, later Google as well. The way to do so was to send them all installs. So the MMP service for developers was, they’re going to deduplicate everything, and they’re going to show you who was the last one to touch.

Now, from here to becoming an absolute truth, this is the source that generated my install, my conversion. I think that that was a stretch that the industry, for some reason, just accepted. Everybody just accepted it. Now, it doesn’t make sense because if you think how marketing works, there is a funnel and AIDA, that’s short for awareness, interest, desire, and action. So usually, it takes some time for users to get to know the product, maybe explore it, click, download, use, spend money in it, and so on.

I don’t doubt the attribution companies’ place in the industry, but I think that last touchpoint attributions simply did not allow you to understand the incremental value you’re getting as an advertiser.

Shamanth: Yeah. And you make a good point about the last touch being very easy to game for anybody that wants to be unscrupulous, including, I would say, Facebook and Google. You know, it’s funny, I was having a conversation with somebody where he said, look if an ad tech company claims the last click or the impression it’s called fraud. If Facebook does it, they call it self attribution. But it’s essentially the same modus operandi. So certainly, I think the last touch can be problematic in very many ways. And you would think that the solution to the problem of the last touch is multi-touch attribution.

Now, with that said, very few companies have adopted multi-touch attribution. We had an episode of the show, where we talked with Holly Chen from Slack, and she talked about how and why slack adopted it. In fact, they were like, oh, we are growing like gangbusters on mobile, but on mobile LTV is so low. And that was just because the last such model wasn’t working. So they built out a multi-touch model. With that said, why would you say nearly every other company has not built out multi-touch models?

Maor: Yeah. So I think Slack, by the way, is a good example of a unique advertiser who’s B2B. I guess a lot of the usability is on the app, but the registration and payments probably happen mostly through the web. Now, when your conversion point ends on the web, you’re much better off trusting the web funnel because, again, they own the domain. So they’re always able to understand where the users have been before they came to their website. They know the funnel, they’re really able to actually control both tracking and then create an attribution model from themselves. Now, again, when you have a website, and when you funnel users convert through the website makes a lot of sense.

When you finally use converts via mobile, this is where you cannot do multi-touch. And actually, when I was working on the startup that I’m building now, the original idea was multi-touch. The original idea was we’re gonna go to advertisers, developers, we’re gonna build attribution as a service for them. So it sits on their domain, and then we can get all the data.

And of course, you cannot really get all the data, Facebook, Google, Amazon, Snapchat, Twitter, Tik Tok, Pinterest; all of these are self-reporting, self attributing. And now, of course, with this iOS 14 change, it’s like, again, it changes everything. So I’m very glad we didn’t do it because it’s not really possible to be able to offer multi-touch attribution in a mobile environment.

Plus,

because you have these behemoths that are self-reporting, the amount of overlap you’re gonna see is close to 100%. As long as you have a product that is mass-market, everybody’s gonna touch the same users. So, no one talks about the reach anymore, but Google and Facebook, they have the same reach. Amazon has a pretty significant reach as well. And so, when all of them are basically claiming, well, we show the user an ad, we generated the install, you have a little bit of a challenge to actually try to even apply multi-touch.

But one way to do multi-touch in mobile would be communism, take your budget, divide it by the number of vendors you have. You get the point?

Shamanth: Yeah, just so I understand this more clearly. You’re saying, look, the touches might happen on Facebook, on Google, on Snapchat, on Tik Tok. And you don’t have access to that data, or you have access to data as reported by them, which is not reliable enough to form a multi-touch model.

Maor: I would say the first. So I think the data is probably reliable. Like, I think that there is a reason why these platforms are so good for developers because they have a crazy reach and crazy targeting abilities, but they overlap. So you’re going to basically end up seeing almost all the time the same user is being touched by all of them. All of them have a reach that is close to 100%.

Shamanth: Right? So theoretically, isn’t that a problem MTA is supposed to solve? Because everyone has the same reach. But a user saw Facebook first, Google second, Snap third, let’s allocate the install and the spend between these three. That’s the problem that MTA is supposed to solve.

Maor: Sure. But that’s the data, you’re not getting on mobile. So you’re not getting impression-level data, or click level data from Facebook. I think from Google, you’re able to get clicks, in retrospect. But when you have just Facebook by themselves, if they’re not disclosing this, and you are a big advertiser, that’s 30-40% of the market that you’re just missing out on.

Shamanth: Right. So you’re getting aggregate impressions and click data that are not individual user-level data. And therefore you couldn’t do this.

Maor: So what is clear is that the attribution companies today take any of them, they are not getting this data from Facebook. So the only thing they do actually don’t get anything from Facebook, other than Facebook telling them, here’s the last time we saw this user, so we claim this install, whether you like it or not. Attribution companies will say we look in their system and say, “Well, no, it was Unity, who generated the last click. So Unity wins the install on our platform.” Facebook doesn’t care. They will still claim the install.

Shamanth: Right.

Maor: And that’s the kind of challenge with this model. That’s why we decided not to go for a multi-touch.

Shamanth: Yeah. Certainly. And again, just to dwell on that a little bit more before we go on – you said on the web, it’s less of a problem, and if I understand that aspect correctly: if you own your domain, you will get individual user click data and impression data. Is that what happens?

Maor: Well, let’s take Facebook. Facebook click from the web goes to your uber.com or slack.com. It’s direct access. It’s not to the app store where you need to triangulate the data, whether it’s IDFA matching or fingerprint matching, you’re basically relying on a third party to kind of triangulate the data and tell you, this is whoever did it. When you own a domain, you don’t need anyone for it.

Shamanth: Right. So you could basically say, user ID number 25, came from Facebook because you had a tag with it?

Maor: Yeah. You set up the URL on Facebook, and you set up the URL to your domain with your parameters. So you have full control, and that’s only on the web.

Shamanth: Gotcha. Now, I see exactly how that works. So, clearly, for those reasons, MTA can be challenging to make work on mobile. Another way to measure performance would just be to measure incrementality. Or to understand what the lift is from any element in the media mix. Can you describe at a high level? What is this approach? And how a marketer might adopt it in a post iOS 14 paradigm?

Maor: Yeah. So In my view, by the way, what Apple did is good for the industry. I’m an opinionated person, many people, of course, think differently. But the fact that Apple eliminates this option of user-level tracking without full user consent, with a pretty aggressive pop-up and their alternative of SKAdNetwork that works on an aggregate level, campaign level and not individual user click install. In my view, it’s good because

marketing is not deterministic. I wish it was, but it’s not. And often, we want to tell ourselves that it’s very deterministic. We know exactly what generated or caused this install. But it’s not causality that we often know. We know correlation.

Often, you know, we run a bunch of ads, we see a lot of installs. It’s fine as long as you work with one vendor, as you can very much understand your incremental value. But what happens when you work with 10, 15, or 60. And the average marketer works with between 15 and 60 different vendors. Now add countries, add various apps; you start really getting lost in the data.

And when you’re only relying on the last touch-point attribution, you’re just seeing reports that tell you whoever touches users last, but you start losing your grip on – is this actually valuable, the spend? Would I have gotten these users for free? And often what causes advertisers to realize that maybe we’re spending a little bit too much, is something that is not related to marketing. So needing to stop the campaigns because you have some issue or bug or you’re running out of budgets, or in the Uber case, it was the Delete Uber hashtag in the US that made them stop advertising.

I’m pretty sure now, by the way, with this pandemic, many advertisers have paused advertising for budget management, and I’m not talking about the travel industry, which is massively hurt. But if you’re an advertiser who stopped advertising for budget management, suddenly you look at your KPIs, and you might be thinking, hmm, my numbers should be significantly lower right now, but they’re not. And yes, there is a halo effect, and yes, there is an organic effect from your advertising. But now that you will reactivate your campaigns, maybe you’ll do it a bit smartly.

Now the challenge is that doing incrementality testing sporadically once in a while doesn’t really allow you to continuously optimize what you’re doing. I know one advertiser, a very big gaming casual company, who does this test once a year stops all advertising; reactivates vendor by vendor, and they do manage to cut away 50% of their budget every year by doing so.

Now, just imagine if they could be doing this on an ongoing basis.

And if they don’t, they can actually now do it better for the next time, and they do get better and better and better. But none of them tries to conclude that what they’re doing is deterministic.

I think that’s a little bit of the arrogance we have in digital advertising because we’ve been in digital advertising, and for us, offline is what we see on Madmen.

Shamanth: Yeah. And I think determinism has always been imperfect. In fact, we had an episode of our podcast that went out yesterday. I think one of the things I said is that LTV has always been a made-up number. And even if you see it as deterministic, you’re making up numbers. I have been in fairly big companies where people fought about how to define LTV. And it just came down to corporate politics, right? Somebody was like, Oh, we need to take two years of LTV, we need to take the ‘k’ factor, we need to take a 50% k factor, we need to give a big launch budget because we have an IP behind it.

So there’s still so much subjectivity. I think it was just a very imperfect metric, to begin with. So suddenly, it makes sense as to the approach you do describe which is, what is the incremental lift we can see from any campaign or any initiative or a channel. So you did say on mobile, some of the bigger companies run anywhere from 15 to 60 channels, multiplied by countries, multiplied by apps.

Now, with that sort of a media mix, how would you recommend that a marketer would think about isolating variables that have an impact versus those that don’t. So it’s like out of 15 if you stop one, you may or may not see a lift. So how should they think about this? How should they think about modeling it if they’re sitting down with their data scientists in the spot?

Maor: Yeah. So I’m a little bit, kind of leaning towards what we’re doing. But I would say that the way incrementality testing works today, Facebook offers incrementality testing. Many many DSPs, specifically in the retargeting area, offer incrementality pricing even.

Now, if you’re a single platform, as long as you have access to user-level data, you can create control groups. You can separate them and show them PSA, ghost ads, whatever. Challenge is you don’t know what these users see, outside your platform. So I would say that you know, the incrementality testing tool is a great sales tool for Facebook. Same as for most DSPs, because you can actually show your customer – “hey, look, I am adding value for you on my platform based on what I can show you.” But indeed, you still don’t know if the spend itself is actually adding incremental value.

Now, our approach, in what we’re building in our company, is we’re using causal inference. So we’re using algorithms that basically look at the differences and changes. Change is actually the most important parameter for us. Because with change we can start creating these branches in time, in retrospect, to come up with recommendations. We’re not claiming to be deterministic, because I simply do not believe in determinism in marketing. But yeah, that’s kind of the approach.

I know that many customers that we spoke with while interviewing to understand what we should be building, many of them tested or tried to do incrementality tests. But either they end up with correlation. Like, you know, it’s hot outside, so I have a fan on me, amazing. But they don’t reach causation. And reaching causation or causality is way more important, and difficult.

And of course, I think many are still trying to do it with the notion of real-time and the notion of determinism. I think Apple forces everybody for a paradigm change, which will take some time for people to adapt to, because everybody’s still used to how things were, and that’s never gonna happen again.

Shamanth: Yeah. And I think making that shift for the first time is going to be a challenge. But people will hopefully get used to the new way of doing things soon enough.

So, in an incrementality model or paradigm, how should a marketer think about small changes? For instance, again, if they have 25 channels, they’re like, “Oh, I want to test Tik Tok on a very small scale, maybe $500 a day to see if it works. That’s the model that has been used until this point. Is that model going to go out the window? Or how should they think about testing initiatives before scaling that?

Maor: Yeah, so let’s, for example, talk about creative, because creative changes sometimes are very subtle, but they mean a lot. And now there, for example, where the last touchpoint is a fantastic proxy. Because the conversion rate between clicking and installing based on the last touchpoint in the same vendor means a lot. And if you have two creatives and one of them is 1%, and the other is 2%, that’s a 100% difference.

So I’m very glad that Apple actually gave people the SKAdnetwork, which is a deterministic way to do the last touchpoint attribution. It sucks that they didn’t yet add creative ID for some reason; it was completely ignored. But I’m fairly certain they will add it later. But again, last touchpoint attribution is fantastic as a proxy for real-time attribution. If you’re doing programmatic and you’re buying on CPM, and you’re spending quite a lot of money, last touchpoint attribution is your best proxy to know – am I actually wasting money completely? Or am I getting something? It still doesn’t mean that you shouldn’t, in addition, look at your incremental lift.

So you know what we’re building is not coming to say, replace attribution companies. We’re not there. We don’t think we’ll ever be there. I think that there was a blind spot. And this was value measurements. And I always use the Uber example because Uber is a public case that everybody knows. And, and it was massive, and they did everything supposedly right. But they still got it wrong.

Shamanth: Yeah. Totally. And if you are advertising only on one channel, let’s just say on Facebook, is the incrementality approach helpful. Or not? Or if so, how would you recommend somebody use it?

Maor: Yeah, well, I would say that it, of course, depends on your scale and size. But let’s say if you’re starting up and you’re running a 20 k a month spend, and it’s all on Facebook, in a single country, single campaign, you’re optimizing for creative – I would still do a sporadic check here and there. Stop the campaign for a few days. Because if you’re spending that little, it’s not that significant, ideally. I would still do sporadic checks, just to constantly optimize how much you’re actually getting out of this spend. But of course, it’s not as important for you if you’re running multiple platforms.

Shamanth: Yeah, definitely. As the number of your platforms and channels increase, I can say your budgets are much larger, so you have much more at stake. And suddenly, this becomes much, much more significant as you grow your media mix. Maor, this has been very instructive, and definitely I think we are on the cusp of some massive changes.

And I certainly learned so much from our chat just now, much as I have from reading your writing style so far. This is perhaps a good place for us to start to wrap up. But before we do that, could you tell our listeners how they can find out more about you, and everything you’re doing?

Maor: Yeah. So first I’m on Mobile Dev Memo, I’m pretty active there. LinkedIn as well, as Maor Sadra and Incrmntal.com, as well, minus two of the vowels in the middle, doesn’t make it easier.

Shamanth: We’ll link to it, so people can just click and go there. Oh, yeah. Yeah. Excellent. So we will link to incrmntal.com. And of course, your LinkedIn. Would you like to tell us a bit more about what incremental is and what you guys do?

Maor: Yeah, sure. So we’re a pretty young company. We actually founded the company only this month, but we already started full time on it since April. We’ve been researching the topic for a while. So what Incrmntal is a software company. It provides testing tools for incrementality for In-app developers and advertisers. Again, we were not counting on Apple doing this change and putting us in a more hyped positioning. But, indeed, now incrementality testing is exactly what many marketers would need with this change. So yeah, I’m glad that we’re doing this because it solves a big problem that has been aching me for a while. I come from 20 years of media. So I’ve always wondered, am I actually giving value? And now I’ll be able to answer this question.

Shamanth: Yeah, Indeed. And you’re going to be able to answer this question without having anything very deterministic. Right. And which is what I find very impressive about everything you’ve described and the approach you described in today’s interview as well. Maor, this is a good place for us to wrap. Thank you again for being on the Mobile User Acquisition Show.

Maor: Thank you very much. Have a great day.

A REQUEST BEFORE YOU GO

I have a very important favor to ask, which as those of you who know me know I don’t do often. If you get any pleasure or inspiration from this episode, could you PLEASE leave a review on your favorite podcasting platform – be it iTunes, Overcast, Spotify or wherever you get your podcast fix. This podcast is very much a labor of love – and each episode takes many many hours to put together. When you write a review, it will not only be a great deal of encouragement to us, but it will also support getting the word out about the Mobile User Acquisition Show.

Constructive criticism and suggestions for improvement are welcome, whether on podcasting platforms – or by email to shamanth at rocketshiphq.com. We read all reviews & I want to make this podcast better.

Thank you – and I look forward to seeing you with the next episode!