Andrew Covato - growth and measurement

This week on the Madvertising podcast, we talk to Andrew Covato. Andrew is the Founder and Managing Director of Growth by Science, a consultancy that helps marketers and ad platforms build truly scientific growth and measurement programs. With more than 15 years in AdTech and MarTech at companies like Google, Meta, Netflix, Snap, and eBay, he’s become a go-to advisor on incrementality, media mix modeling, and how to assess the real ROI of advertising in a privacy-first world. 

Transcript

Adam Singer (00:00):

Hello listeners and welcome to the latest episode of the ad Quick Advertising podcast, A podcast that explores the intersection of technology, content marketing, all of the things that you care about. Today we have a special guest. We have Andrew Covato. He is the founder and managing director of Growth by Science, a consultancy that helps marketers and ad platforms build truly scientific growth and measurement programs. He has a long storied career. He's been at Google Meta, Netflix, snap and eBay. So companies no one's heard of, right? He's a go-to advisor on incrementality media mix modeling, how to assess ROI, and he really is an OG in the marketing and analytics space. Really smart guy. We're lucky to have him on the show. Andrew, welcome to the show.

Andrew Covato (00:48):

Thanks, Adam. What an intro. I feel like I needed some music behind that awesome intro. I really appreciate it and pleasure to be on here with you guys today.

Adam Singer (00:56):

Yeah, someone mentioned that we should do music for the show, so it's something we might do in the future, but we're just talking to marketers and advertising professionals while they work and we're very informal and scrappy and I think that's one of the things that people like.

Andrew Covato (01:11):

Love it.

Adam Singer (01:12):

So you've worked mentioned across Google Meta, Netflix, snap, really great career on awesome companies and now run growth by science. And so I guess the first thing I'd like to ask you because seen so much is what are some patterns in how you saw the best organizations and the tier one companies you were at and how they approach measurement versus everyone else who maybe isn't an ad tech and doesn't have an insane data science team?

Andrew Covato (01:43):

So let's break this down into marketers and ad platforms. I think measurement's very important to both of those industry constituents and they obviously approach it differently. So from the marketer's perspective, I think it starts with a healthy dose of skepticism, a culture of experimentation and a true desire to bring the principles of science into all aspects of the business, including marketing. And so that's maybe a little bit fluffy a little bit, but really and truly, companies that don't have that tend to be attracted by the easy way of doing things in marketing measurement versus the right way of doing things. And so when you have that culture, I think Netflix, the experience that I had working there is a great example. Netflix is very scientific, very soliciting descent and trying to peer review internally. All the different ideas. Companies like that tend to want to get the right methodology is in place to truly understand what their marketing investment is yielding from an incremental causal perspective.

(02:53):

And obviously there's a plethora of approaches that one can take to that, but most of them are centered around some type of experimentation, a test versus a control and some type of modeling, whether it's an MMM or other type of causal model that work in harmony with maybe some more real-time attribution type data. To give you sort of that triangulated perspective on what marketing investment is yielding to the platform in terms of ROI. So that's on the marketer side. On the ad platform side, there's sort of a strata of different, or there's strata of different marketing platforms and each of 'em approach measurements slightly differently. At the top, you've got the big dogs, you got the Googles and the Metas of the world who are commanding large portions of most advertisers budgets and they're trying to shape the measurement narrative themselves. And so they want to understand what types of methodologies and approaches are not just good for outlining ROI for the advertisers, but also beneficial to them and allow them to paint a really good go-to-market for their ad products.

(04:03):

And so a lot of that involves building some very innovative first party solutions, having unique partnerships with big third party measurement platforms that are out there in the ecosystem. And so they kind of set the agenda and set the status quo, which can sometimes overlap with the interests of the advertisers, but not always. I like to say that ad platforms are not in the interest of optimizing for outcomes. They're in the interest of optimizing for their own revenue. And so obviously you can't not deliver outcomes for the advertisers, they won't pay you, but if there is a way to extract more and to capture more of those advertising dollars, they're going to be centered around that approach. So that's the top guns. And then the more incumbent platforms are really almost at the mercy of the advertisers. So what measurement is important to advertisers, they're the ones making those demands of the platforms and the platforms have to, whether they agree with it or not, oftentimes develop methodologies or make those partnerships so that they can woo the advertisers to their platforms. So it's kind of a mixed bag. It's definitely, I would say continues to be a little bit of the wild west and measurement land. In 15 odd years that I've been working in the space, the same kind of challenges continue to be prevalent and there's definitely been innovation, but the challenges do remain the same.

Adam Singer (05:34):

And so you've had a really great career working for the top percent companies. Was there a specific moment in time that you decided, you know what, the industry needs a more scientific approach to advertising measurements and it was enough for you to start your own company around that thesis?

Andrew Covato (05:54):

I feel like even while I worked at these companies, I tried to, and I was always client facing for the most part. I always tried to describe an objective scientific approach to marketing. Measurement is just sort of the way I was wired and wired. And so what frustrated me was the lack of, how can I say it? I guess the lack of proactivity that existed on the advertiser side to jump on some of these. It almost seemed like I was giving 'em the right answer. And they're like, ah, but it's so hard. I don't want to do that. I'd rather just stick with last click because it fits nicely into my models and I can get all of my projections done pretty easily. And so there's definitely a bit of that complacency that sort of frustrated me. But over time, and I think now especially the last few years, things have changed where there's more of a need on the advertiser side to get really scientific about what they're doing.

(06:56):

And so really the impetus for growth by science is less about, I would say bringing science to marketers. It certainly is that, but it's more like the way in which we're doing that. The gaps that I've always seen in measurement are every company has some unique challenges. And so there's a lot of great SaaS tools out there, third party measurement tools, most of them are so restrictive or they don't provide the depth of customization that especially more enterprise advertisers require to solve their measurement challenges. We kind of fill that gap. We kind of take the best of both worlds, the nimbleness and the agility that the SaaS platforms provide with more of a hands-on bespoke approach, we can kind of tailor our models and our approaches to the different nuances of every advertiser that we serve. And so that was really the main impetus is there's this unserved segment of the market of where folks are like, yeah, this is so good and I wish that this SaaS platform could just tweak it a little bit. And there's that last mile of integration or customization that was lacking. We sort of fill that void.

Adam Singer (08:07):

So you talk a lot about incrementality and scientific growth programs for A CMO or growth lead who still lives in platform dashboards. How do you explain incrementality in such a way that you can get them to actually change their behavior?

Andrew Covato (08:24):

Yeah, it's a great question. I would say the first step is understand kind of something that I said before. Platforms are out there to optimize their own revenue, not advertiser outcomes. So just go in with that level of skepticism. And they're not doing it on purpose. They're not trying to intentionally trick anybody, but that's there a for-profit company. And so obviously that's what they're out there to do. And so go in with that apriori mindset when you're looking at platform metrics or anything like that. And then ask yourself the question, what would happen to my growth if I shut this channel off? Or if I shut off all of my marketing, what would happen if you don't have a very clear answer on I would lose this number of conversions, then the only way to answer that is with an incrementality test, a lift test, a holdout test of some kind or some way to assess that. So it's really understand that revenue optimization is the platform's top priority and really have that idea in your mind of what would happen if in a world where I was not marketing this way, where it did not market on these channels, what would happen to that? Have a very concrete answer to that.

Adam Singer (09:40):

And so another question about maybe not even just platform measurement, but analytics tools. And I feel like a lot of marketers, hopefully most marketers understand not to use something like last click attribution by itself. That's obviously going to bias attributing,

Andrew Covato (10:00):

You'd be surprised.

Adam Singer (10:01):

Well, yeah. I guess what are some of the ways that you see just any analytics tool get attribution wrong last click or I think there's probably some more interesting answers just around attribution more broadly because there's linear, there's all these different models. Why as a marketer shouldn't necessarily just trust this? And I guess even more broadly, just how should I think about attribution given to me by an analytics tool?

Andrew Covato (10:34):

Yeah, I think attribution, I feel like is a very loaded word in my mind when I think of attribution. It means the kind of old school MTA style where I've got a path of touchpoints that I've linked together and I stitched that to a conversion event that matters to me, and I'm going to use some kind of business logic or math to distribute credit across those touch points. So that would be my definition of attribution. And from that perspective, we at growth by science do not do attribution. And there's certainly a lot of measurement SaaS platforms that do not do attribution in that way. So if a marketer's idea of attribution is that kind of path to conversion type dataset that's required, I think they will, I, I know they will be missing the true incrementality of those platforms. Just because you have all of that data, there isn't a way for you to understand what would happen in absence of some of those marketing channels or one of those marketing channels being present.

(11:39):

And that's where you get incrementality. It's the idea of a counterfactual, right? The idea of a world where an equivalent world where everything's the same except for one thing being different and have to able to, the best way to do that, and I won't say the only way, but certainly in our opinion, the best way to do that is with an explicit design experiment where you've got a treatment in one group and you've got a control in the other group and you can understand what that delta is. So I think one of, to answer your question more directly, what do folks get wrong when they look at say, an attribution type dashboard is I think they take for granted that two things. One, that the data is all correct and that the paths are correct. That's never the case. There's so many reasons why you cannot get that perfect path to conversion data set. That's number one. And then number two, assuming you did have perfect data, there isn't the right kind of math behind it to give you an understanding of how to distribute that credit. It's all kind of hocus pocus. And so when we come in and when we work with advertisers that have some kind of MT Aish attribution going on, it is inevitable that we find significant pockets of over or underspend just because the attribution tends to overvalue or undervalue channels disproportionately.

Adam Singer (13:02):

That's a good answer. And so we've talked a lot about things that I think are for probably more well-resourced, larger marketing teams. I'm curious if I'm at a scrappy startup with just a few people and I'm a marketer and I basically am not doing anything sophisticated yet with MMM or anything like that either because I'm not staffed for it or I just don't know, what would you say someone in that position could do to start to uplevel their marketing? What are some simple things? Is there a simple tool you recommend? Is platform data good enough to get going? What do you typically advise marketers that are just super resource constrained?

Andrew Covato (13:51):

Yeah, I think if you're a very early stage company that's the new brand in a certain space and you've just started marketing in a somewhat meaningful way on say some digital channels, you can probably get away with platform attribution. And I would say that's the only special case where it may work. And the reason I would say that is because you've almost got a natural experiment where you were doing nothing and nobody knows anything about your brand, and then all of a sudden your brand is live. And so if you do see a commensurate jump in conversions, it's probably the ads, right? You don't need to do a fancy experiment to figure that out. But over time, that's going to change that sort of transient jump that you see when you just turn your ads on. It's not that jump is not constant over time.

(14:46):

And as your ads become better or worse, that delta changes. And if you're not periodically checking back what does it look like without ads, you're not going to understand how that delta is. So I don't think that really resource constrained companies who are maybe investing on just a few channels, say Google search and Meta and TikTok, something like that. The easiest thing for you to do if you feel confident enough in doing this would be do a simple geo test where you shut off ads across all of your media in a couple of geos, make sure you've statistically balanced it appropriately and whatnot and understand what that delta is. You don't have to do this all the time, but maybe once a quarter, once every six months for a couple weeks or two to four weeks, shut these off and see what happens. Now the pushback I get from that is, oh my gosh, opportunity costs.

(15:36):

I'm going to give up growth. Yes, you absolutely will, but you don't have to. That money that you were going to spend doesn't have to go away. You can catch up before or after or reallocate it to the on, so you can keep things in a budget neutral capacity, number one. Number two, what's the opportunity cost of badly measuring your marketing if you shut off? So you have three channels, you shut 'em all off and a few geos and nothing changes. I mean, wouldn't you like to know that, right? There's not an opportunity cost in that sense. And now you know, okay, what I'm doing isn't working. I got a hard reset and fix it. Alternatively, you could shut it off. You could see, wow, there's a massive delta going on here, in which case you've been leaving money on the table and you wouldn't have known that if you hadn't have shut these off.

(16:28):

So I think that's the most powerful thing that any company should do is, and we advocate this even for our largest brands, is shut off all your media. Do it in a smart way and the art, and the reason why we're trusted by the large enterprise advertisers that do trust is we have ways to do that to maximize the signal that we can get out of the experiment while minimizing the amount of time that the need has to stay off. And we can keep things budget neutral and all of that. So when you get more advanced their complexities, but the same principle applies whether you're spending 10 KA month or whether you're spending 10 million a month.

Adam Singer (17:04):

I always found the small teams that I've done marketing on, first of all, it's easier. You don't have 10 groups trying to claim credit for everyone. It's either your work or not. So it's pretty clear if you marketing is working, and also like you mentioned, it should be pretty easy for you to run simple tests and experiments. There's not like 10 layers of approval to turn something off or try a pretty big change. Like the old adage, you can just do things. I try and advocate that for smaller teams, half the fun, you don't have bureaucracy, you can just ship whatever you want.

Andrew Covato (17:39):

Yeah, a hundred percent. And I think it's worth mentioning what I described is a great approach if you're marketing on traditional channels, but obviously there's a lot of new channels out there. Well, not new, but there's a lot of investment in channels that are harder to do that on off type testing. You've got everything from podcasts to influencer and content creators to out of home. A lot of those channels are more challenging to test than on off capacity, not impossible. And where you can do geotargeting on some of those definitely would lean into that as a primary approach. But you still, in those cases, you have to seek natural experiments where you're turning things on, you're kind of looking at more of a pre-post comparison and then just have to make sure that you're applying the right math to those scenarios.

Adam Singer (18:37):

Thankfully, there's an ad tech that makes the out of home tests fairly straightforward.

Andrew Covato (18:43):

Yeah, yeah, absolutely.

Adam Singer (18:45):

And hint, shameless self plug,

Andrew Covato (18:48):

And I would just say from my perspective, I think that's a huge powerful differentiator when you can do the geotargeting on any channel. So definitely worth looking into if at a home is a serious investment for you,

Adam Singer (19:04):

The geotargeting stuff is just so smart. It is just such a great way on whether you're doing stuff in physical world or TV or out of home, you should really run those tests and then when you scale up, you already know it's going to work Well,

Andrew Covato (19:21):

Without question. The interesting thing, Adam, is I do know of a number of surprisingly large advertisers that will not invest in a channel if you cannot have certain targeting capabilities geotargeting. And most of it is for measurement purposes. And again, these are top, top multi-billion dollar advertisers that will not invest in channels if they lack certain capabilities that allow them to be measured by their own paradigm. So it's something to keep in mind. And even as a smaller advertiser, I think you should have a strong opinion about how you want to value your media, and if a channel's making it hard for you to do that, you don't have to live and die by any given channel. There's always alternatives that are out there.

Adam Singer (20:11):

Totally. So Andrew, you've been vocal about the dangers of AI and ad delivery when it's optimized for the wrong objective. I think I took notes on that right when I was doing research for the show. So as platforms become sort of more automated and opaque, what should smart advertisers be doing differently or look for to keep control of their performance as AI advances and tries to do more?

Andrew Covato (20:37):

Yeah, little bit. The train I think is left of station to a certain extent. I mean, we're seeing the right on the wall. All platforms are eventually going to reduce controls, but I think any advertiser should be intelligently testing any kind of AI platform. There's a number of options out there. We have helped advertisers test into some of those AI based tactics, and they've been a mixed bag, if I'm honest. Some of them have worked really well for some advertisers. Some of them have not been as incremental as some of the more manual approaches. Now, the optimist in me believes that eventually those types of campaigns will become really, really good, and I think they'll get really smart and understand how to deliver incrementality, but we are not there yet. And I would just say, let's be forward thinking and let's try new things and lean into the AI campaigns as marketers, but let's test them and let's hold the advertising platforms accountable for when they're not delivering what they need to by shifting dollars away from those strategies to other ones.

Adam Singer (21:42):

Yeah, it's been really interesting watching, and this sort of predates AI with Google kind of forcing P max on people where

(21:51):

The ad tech platforms grow and it's like the first products were really good and let advertisers have all sorts of control, and it's just sort of this slow claw back with a lot of the big tech companies, like you mentioned, they're optimizing for revenue, not necessarily for the user. And the user in this case is the advertiser. And it's just always so interesting to me because when we started, it's almost like some of the tools have gotten a little worse and better at the same time, but worse in terms of, I feel like in a lot of the product meetings, they just don't assume you're a power user. They want to assume everyone's the same. And it's been an interesting progression watching how some of these ad techs have worked. I think Facebook is another one where not only have they given less control, but the product just perennially is slow and is annoying to work with. And I feel like it's sometimes two steps forward, one step backward with a lot of what these companies do.

Andrew Covato (22:52):

I think you're right about that. And honestly, I think we're still in the phase. It'll get worse before it gets better. And so we kind of have to suffer through it a little bit as an industry and again, just keep our scientific measurement approaches strong, make sure that we're not over-investing in things that aren't working and just keep that approach very, very rigorous. But I do think they're going to get better. I do think that there will be a much better call it per advertiser optimization and personalization based on each advertiser's specific need and campaign within that need and all of that. I do think we'll eventually get there, but it's a matter of when and how much wasted dollars advertising dollars do we have to get along the way. And so the onus is ultimately on the advertiser. I say this all the time, you've got a lot of options where to put your dollars and how to spend them. And it is up to you to have a set of principles and some valuation on those dollars so that you know where and how to spend them.

Adam Singer (24:02):

Agree, totally. So Andrew, you've had a very storied career. I think you've worked at companies that are harder to get into than an Ivy League college, several of them. And so a lot of young people are definitely struggling right now, especially as they're entering a workforce that's increasingly powered by ai. But I'd love to ask you, since you've been successful getting to work at some of these brands, what would you tell young people starting their career today who might want to even work anywhere in tech, what they should do to really stand out and get to the table, to have some conversations and get a shot to work at a fang?

Andrew Covato (24:41):

Yeah, it's a great question. And honestly, I mean, I feel I can share some thoughts, but my FANG career is a number of years out of date at this point. So I would imagine that folks more close to it now might have a different opinion, but at least from my perspective, I do think having a solid background in some kind of quantitative approach, and even if that's not necessarily what you studied in college, supplement that in some ways to make sure that you have an understanding of math and math as it applies to business and to marketing. If that's thinking more on the marketing side, I still think that there's an edge to be had for folks that have an appreciation and understanding of that. I think you have to be AI savvy in this day and age. I would be hard pressed, and I would imagine that almost every tech interview, there's some question about ai.

(25:37):

So I'm not saying you're necessarily writing your own a agentic code or you're building your own LLM or anything like that. You don't have to be that crazy, but have an understanding of what it is, how it works, how it applies to your business, what kind of applications you could see for AI in the future, things like that. And then in the marketing space, I do think kind of going away from more of the hard sciences, I do think there's still a massive need for a really good story, a really good pitch. Ultimately what is going to make the ads work is a really good story and a really good pitch. And so having some intuition and some understanding about how do you connect a bunch of users to a prospective product or service, what are things that drive that connection that I think is a place where AI will probably get to eventually, but is lagging in the current time. So I mean, I'm not sure how helpful that is to be honest in how fast things are changing. I'm a little mystified into what kind of skills somebody entering the workforce might need, but if I had to make an educated guess, it'd probably be what I described. But every situation is different. So

Adam Singer (27:00):

Yeah, there was just a story in the Wall Street Journal basically saying companies were looking for storytellers. And it's funny because every few years there's always a new rebranding of marketing and companies have to seek out someone who's good at SEO, then social media, then the metaverse, now ai. But to your point, the core of, I love what you said about the core of storytelling and math. There's a story I'm reminded of. I was in Vegas once at a technology conference and I made friends with the director, the BI director at Caesars, and he gave me a behind the scenes tour of the business department at Caesars. It's really boring, by the way. It's not like the rest of the casino, it's just boring corporate offices. And I was in his office having a coffee with him, and I saw on his desk there was a resume for a lady who had a background doing pharma, and she had done all this oncology cohort analysis and whatnot, and I'm like, why do you have someone from a biotech background interviewing here?

(28:06):

He's like, oh, they're really good at data science. We can train them up to understand all of our business problems. And I'm like, can we keep some people on the oncology stuff and maybe not figuring out price elasticity of table games at hour per day? And he laughed and he is like, well, we're a good employer. People get experience. And it was just really interesting. But to your point that there's not enough people who are really good at these things and companies will source talent. If you have a background and you're a little ambitious, I think it's whatever you can get on your resume.

Andrew Covato (28:40):

Totally. And there's a lot of different strategies to try to work your way in tech. There's take a bigger role for lower pay at a smaller company or take a smaller role for maybe bigger pay at a bigger company. And I don't think there's a right answer either way, but my approach when I was deep in job seeking and career building mode has always been take the coffee, be open, put yourself out there a little bit, and sometimes you'll have some awkward encounters and something that is totally amiss or you misunderstood something or they misunderstood something about you. But ultimately being open and receptive to opportunities that come your way is the best way to get ahead. I'll be honest, I did not think that I work in marketing analytics. My undergrad was in engineering and manufacturing engineering, which ironically might be useful nowadays, but it was that, and I did a master's in finance and I kind of stumbled into marketing analytics and tech and that kind of thing. And so it wasn't a forethought, but the careers kind of gets legs of its own and can sort of take off on its own if you're open and if you're interested and curious about what's out there for you.

Adam Singer (30:03):

Yeah, John Miller is someone else. He has a background in physics and he ended up being the co-founder of Marketo. So I really think a lot of the very analytical brain people have played an outside outsized role in building marketing over the last few years. I think there'll be some return to the mean and some more demand for the creative types. And ultimately we need both, right? We need everyone in the room working together for sure. I think everyone is on the spectrum of having really good analytical skills, really good creative skills, very, very few people in the middle, very few. If you're actually good at both, you're pretty much a unicorn. But I find everyone's somewhere on the spectrum and it's a little bit of a cognitive trade off. So Andrew, this has been really interesting for the last part of the show, tell me about the company. Tell me what types of people should give you a call for services and we'll wrap

Andrew Covato (30:59):

Up. Yeah, thank you. It's a great discussion, by the way. I've really enjoyed it as well, and happy to share a little bit about growth by Science. We are a marketing analytics consultancy. We serve marketers who are struggling to understand how to value their media investment. We take a very hands-on approach. We've got a lot of proprietary tech on the back that helps us deliver our services quickly and accurately, and we're there to solve the most challenging problems in marketing measurement companies who are spending, I would say anywhere from 10 50 million to over a billion is sort of our sweet spot. And we've served advertisers all along that spectrum, helping them design incrementality, tests, design and execute those tests, build custom MMS to serve really nuanced businesses, regional differences, hard to measure channels. Basically, if you're struggling to measure it and you struggle to find a fit with some of the tools that might exist out of the box, we should be your next call. We've been there, done that, and we've got a lot of great folks on the team with experience across the most challenging corners of the measurement universe, and we'd love to lend a hand and help our advertisers just spend their marketing dollars more intelligently.

Adam Singer (32:20):

Awesome. Well, Andrew, thank you for the conversation and for everyone listening, we will see you on a new episode next week.