So showing up and then again results as well as kind of paid results. And so I was responsible for much other things that optimizing the Amazon homepage to show up in Google. And so I wrote that I personally wrote the title tag uh of the Amazon homepage to get the right keywords in there. And nearly 20 years later they haven't changed it, it's still the title I wrote. This

episode is sponsored by sweep the floor. A new podcast founded by ex Amazonian Jason ver lindy with sweep the floor. Jason interviews some of the most interesting and quirky makers working today, from woodworkers and mechanics to surfboard shapers to entrepreneurs. He just interviewed ceo and wooden canoe builder. Trent, Pressler, the author of little and often out now and purchase a bill on amazon, find the podcast at sweep the floor dot net. I'll link to it and the book Little and often in the blog post. Hi, I'm Dave chappelle and I'd like to welcome you to the fifth episode of the Invent Like an owner podcast where I talk with the amazonians who helped build amazon dot com into one of the world's most valuable companies. This weekly podcast is for entrepreneurs, business leaders and all students of history. The goal of the podcast is to capture the amazon creation stories and create a historical archive. On that note, my guests are recalling history as best they can.

It's possible some of the details are fuzzy or just plain wrong if that happens, it isn't intentional. I invite future guests or commenters on the site to help us get the facts as straight as they can be Now on with the show. Today, I'm excited to be talking with Blake Scholl Compared to some of our previous guests. Blake arrived at Amazon a little later in 2001 Still 20 years ago, but you'll see that there was still plenty of impact to be made. Blake conducted amazon's first google Adwords by and his team built the technology behind amazon's large scale automated search advertising on google Today this technology drives billions of dollars in annual sales for amazon and I think you'd be hard pressed to find a google purchase intent search that doesn't have amazon in the paid placements today you'll learn about how amazon worked to become the starting point for all shopping searches and how that manifested in various investments and experiments by the company. And you'll also meet Blake who is quickly becoming one of the most interesting people in the world in my opinion, welcome blake.

It's all very kind of, You dave great to be here. This is a, this is a fun set of stories to share it and excited to be talking about

it. Great! So you joined Amazon in 2001, fresh out of Carnegie Mellon and, and it looked like several interesting internships at real networks and ink to me. What attracted you to amazon versus other alternatives? Or was it was it your goal, That

sort of thing? It did turn out to be my goal, but it wasn't my initial goal. I've always known exactly what I wanted to work on, although my view of what that is keeps changing. By the time I was midway through college, I needed ultimately I wanted to have a career as an entrepreneur And I wanted to go somewhere where I'd be able to not just write software but to really understand the business side and be able to contribute that way. And so I remember like it was an Amazon customer since 97. Uh I went to the home page at one point, uh oh this is filled with stuff I love and for like 20 seconds I thought that some editor at Amazon like completely had all my taste in literature and I thought that was awesome, that I realized that it's not that at all the software that's doing this. And I start to think, well how do they do that? And so I fell in love with the kind of the automated merchandising personalization technology. I decided that's what I wanted to work on after school and I started restricting who built that. I interviewed with a company called Epiphany and I was like, oh do you guys do this stuff on amazon then,

no, and eventually realized that the amazon was the place, this is done in house. And I got to snag an interview. And so this was late 2000 and so the dot com bubble was kind of imploding at that moment. And my friends would say like why are you going to work for amazon? It's a bookstore. And like do they even exist anymore? But everyone I met there was like super smart and down to the engineers who interviewed me, I could ask them business questions and they can explain cash flow cycles, operating cycles, Everyone else says this is going under, but everyone I meet there really knows what they're doing. And so I went to amazon to be a software engineer working on personalization

software. Was matt round leading the team at the time or was it still josh?

I think it was right about the transition from josh to matt at that point. I think josh was leaving when I interviewed and matt was doing around the time I started,

Well I'm we're hoping to talk or interview both of them together in the next couple of weeks. So that's been top one to get just because it's such a good

story. That's gonna be a great one. They're both fascinate people to talk to as well.

Yeah. So tell me about your first project then. So you were hired as a software engineering on the personalization team. Where did you spend your first, let's say a year or two? Was there one big media project that you started off with?

No, it's a lot of little stuff. The gist of that team in amazon was it was very quantitative and very focused on incremental changes that would drive revenue or profit. And so your typical project, three month project was insanely long. Your typical thing was maybe 3 to 3 to six weeks of development effort. And then you get out there and you run a web lab which was the amazon term for an A. B. Test and if it made money you were happy and if it didn't you learn and iterate differently. And so I worked on a bunch of different things, a lot of it was either related to people who bought this item, also bought or performance of the website and performance, the personalization features. Like I remember at one point I was charged with making the homepage faster, which is in some respects a hard thing to do. We redid some of the data architecture that helped. And then and what we realized at one point was that we render the entire homepage of the website to a buffer and then we were done would send the whole thing and so I ran a little A. B tests that would just basically dumped the buffer after if you think back to then there was the tab bar and so what if we flush the buffer after the tab bar and it turned out that drove another like percent or to a sales just have the tab bar appear faster. And so lots of little things

like that. So when you worked on the homepage, let's call it a speed test. Was that more for shopping? Because it loaded faster people bought more or did it also relate to sort of an early premonition that page speed would be something that google and other search traffic centers would like,

well amazon. I think we had an early recognition that speed mattered, which I guess that ends up being a theme through my career in some ways they had an early idea that that speed mattered. I think at the time there was like a goal that had the homepage under six seconds, which of course sounds sounds a little bananas 20 years later that you'd allow the homepage for amazon to take six seconds to load. And so there was when we went and measured because the amazon what amazon measure everything. When you do one of these A. B tests, you get back a report that's like 200 variables along with statistical data on everything that would happen. And so we sped up the homepage by just sending the tab bar faster. We saw two major effects. One was that people were less likely to just leave. The site has been ineffective. Like you type in amazon if it's just sitting there not doing anything, you're much more likely. So I guess it's not working, I'll go somewhere else.

And then it also changed where traffic went on the website. And one of the reasons that drove increased sales is people would stare at that tab are longer and they'd be more likely to go to the personalization features. So back then there was a whole tab that was called like your store, you would be like Dave store blake store. And that was where all the algorithms would like try their best to sell you stuff you really want. And it was one of the best pages for generating sales on the site. And so getting the tab are out there turned out to drive more traffic into the personalization features, which drove more sales.

It's funny you would almost think you want to load the search box, then load the tab bar or vice versa and then sort of top to bottom load the page if it were possible,

You would think. But this was 2001 or 2002 and Amazon search was not very good back then. So search was not the best place to drive people and a lot of sales you get through quote unquote discovery where people were not necessarily coming looking for one thing, but they're kind of browsing and the personalization futures were really good at getting people to buy things that they didn't know they wanted through browsing experiences.

Was Amazon beating that 6 2nd time before you got involved or how long did it take in your memory to

get there? Oh, my memory is very fuzzy on this. My recollection was that this was a multi team effort that went on for went on for months before it really succeeded. And then once they got to six seconds, the new number was two seconds and the even more effort to get there. But the big culprit, the big culprit in the homepage at that point was personalization because it was, Remember the data architecture of the early 2000s, there was an oracle database as a service that we give you the most recent most recent items that the customer bought. And then there was a whole set of data that would in fact today people talk about it coming up machine learning, but it was like baby machine learning that actually had the personalization algorithms and that was a gigabyte or two of data that lived on literally every web server. So there were maybe a few 100 web servers and Amazon every single one of them had a copy of this data. And this is back when ships for 32 bit. So they all had four gigabytes of memory. In fact, there's more data on these machines that would fit in RAM.

And so you're going out to the hard disk a bunch. And so computing personalized recommendations was something that was very computational intensive. And so the work that we did to shrink that data set and ultimately put it in a service oriented architecture made a big difference to the speed of the personalization features, which in turn made a big difference. The performance of the homepage of the website,

Right. And something that people might wonder about is for a person who visits the site who's never been there before? Was the site faster for them because there was less quote unquote personalization going on or did they have new problems?

In fact, personalization? Was this like this backwards and how you wanted to? Because the more you bought in amazon, the worse it got. Uh in fact from a couple of different aspects back then, the average customer had bought 2.5 items today. People like 2.5 items a day from amazon. And so what would happen was that the more stuff you bought, the more input, there wasn't a personalization algorithm, the slower it would get. Which is that the last thing you know the best customers had the worst experience, which is very possible. In fact I remember tell a funny story. So amazon had this trouble ticketing system and there was this notion of a severity one severity to incident which is like somebody is massively wrong in the business said to is like a big part of the websites broken a sev one is like the whole websites are broken or there's something wrong in the warehouse, you can't ship any packages and there was this page you could go internally,

just see all of them. And I was very curious and I would just load these for entertainment because I was curious what was going on and at one point there's a sev two ticket. This is customer 8573592 Can't load the homepage and we'll play in the world would be a sev two ticket if one customer can't load the homepage. And so of course you click, you click on it and I probably shouldn't say who it was. It was not an amazon employee, but it was well known external individual Who had been the first person in Amazon history by 1500

items. I know who this is by the way, because they were mentioned in a question in the last episode that we had to delete because they were buying things for their library I believe. So

anyway, that's believable. So I won't cause you that same edited chore by mentioning the name but this individual, but more than 1500 items which broke the wire transfer protocol from one of the personalization services and the homepage wasn't leg

awesome. So you spend a year or two working on lots of things. The homepage project. We're going to talk a lot about the automated advertising work you did. How did your role evolved to start working on that. Was it something you signed like you were excited about doing and maybe give some background for listeners about like early internet gatekeepers and like and this problem that you were test with working on over a couple of period a couple of years.

Yeah, I guess you can tell a little bit of my personal story and then I can also tell that the strategic story, which would probably be a little more interesting. So my personal stories before I went to amazon, I accepted the job but I didn't know the first thing about any kind of automated merchandising or automated machine learning kinds of stuff, but I was terrified. I would suck at it. And so the summer before I actually started, I decided like, okay, I wanted to go build just to build some stuff on the side that would get my feet wet with it. And so being a 20 year old, what I think of building, I thought of building a dating site that would match people up with algorithms, you know, like people who look at this profile also looked at and so and so I wouldn't built that for fun. And then I was like,

how do I get people to the site? And that was right around when Google has launched Adwords. So this was summer summer of 2000 and one. Early summer of 2001. And so I figured, okay, well this looks like a cool way to get people to the site. So just playing with Adwords, I got to kind of know the system through having promoted this dating site I was building and then, you know, it's a fast forward a couple of years now. It's, I think late 02 and my manager walks into my office at one point and he's like, hey, there's this, there's this project called like,

see if we can make any money buying ads on google that keeps getting kicked from team to team and somebody gave it to the best of guys and they couldn't for how to do it. And so do you think, do you think our team should work on it? And if so, who should work on it? And I was like, how about yes and me? And so it kind of landed in my lap. And so it was teed up as like, okay, this is all just an experiment. And like I said earlier, it was coming from the context of project is rarely more than like six weeks, maybe a little bit more than that in this group. And so we thought we didn't have a lot of time to go, kind of figure out if they're receiving and they're they're and, moreover matt round, who you mentioned earlier, who would be awesome to have in the show called me into his office, he was my manager's manager at that point,

essentially for listeners was I think Director of VP of Personalization, so he ran the

Director of Personalization. He was he was one of the many people who had followed basis from the shot amazon. And so that's like Jeff cares about your project and if you mess up, he will back or manage your project. And so, so you need to go, you need to go meet with Jeff and basically earn his confidence very early on. So he'll leave you alone to get it done.

Well, the flip side of it is, you can say, well actually I want a lot of time with Jeff, so I'll mess up just enough, you know, to, to get exposure

that I did that, that even tried.

Yeah.

So I go and have my, I'm like 20, 21 22 at this point, uh, have my 1st 1st meeting with Jeff and it's on, it's on friday at five o'clock. And uh, some called into Jeff's corner conference room and I'm like, you know, it's me and my direct manager and Jeff and I'm just like nervous af And, and Jeff looks, and this was back when amazon with a segue had just launched and Amazon was the distributor for Ceg Jeff's icebreaker was hey, I want to read my segue. And so we spent 15 minutes going up and down the halls of the office writing Jeff Bezos's segue before we sat down and actually had a conversation and he was very, very specific about his strategic guidance and he didn't want us to teach anything to google while buying ads and he told me a story about something that had happened in the fulfillment centers. And so as as I think people know the early amazon fulfillment centers were all insourced and then as they were scaling, people start to build e commerce warehouses and so amazon would outsource and fulfillment and what happened is all the fulfillment outsourcing companies,

like they all suck, they weren't as good at this amazon was and what amazon do they teach them how to be better because they want their suppliers to succeed and then one of their suppliers there, they go off for like now higher quality fulfillment services to all of amazon's competitors. And so Jeff said treat google as a mountain, you can climb the mountain but you can't move it. I'm pretty sure that was very close to word for word what he said and so okay, well we'll have to figure out how to buy ads from google at scale without, without really talking to google at the time. I didn't appreciate why in the world is this project I'm working on as like a 22 year old kind of on the side for my team, something that the Ceo is interested in at the time. I think I thought Jeff thinks it's cool, but looking back, I think there was like a massive strategic reason why this got so much attention and this whole thing played out. Like I'd see Jeff sometimes it was like six weeks apart and I have to update all the leadership team, the s team at least once a quarter and how this whole thing was going. So why? Well this was the early two thousands, it was not clear yet how shopping would start on the internet and google was on the riot. And it was a big strategic question is shopping search different from web search. And if it turns out that shopping search is not different from web search then you could expect that people will go to the google search box for everything they might want and then anybody who wants to sell anything on the internet is basically I have to pay rent to google for the

privilege and prior to google just for people that are young and don't remember A. O. L like priority google. There was the ol the yahoo and these places basically charged us rent. Like we had to buy mega deals where we would sponsor the book category, in the music category, in the video category in the early days, that was how amazon got a lot of traffic. But as google be in IOS, let's say an android google became dominant, they became the big gate keeper, whether paid or organic. And so you're basically talking about the top of those google results were mostly

paid. You're right. Like there's basically, where do people start on the internet in those days, it was mostly not amazon. So amazon has to pay, you know, call it 5 to 10% of sales to the gatekeeper for the privilege of having gotten the traffic to the site. And so if there's a new place, say google where a lot of traffic is going to start, you need to get really good and showing up at it, showing up in it. Or better yet, you could have a strategy that people will start e commerce search at amazon, which would be a massive strategic advantage. And so when I look back at it, I think amazon had,

this is no authority higher than me. Looking back 20 years later and thinking about what must have been going on. So this is not gospel. And it might be, if you talk to more senior people, they'd say no blake's crazy. It's a completely different story. But my view of it is amazon had plan A and Plan A was to become the equivalent of the default app for shopping when people wanted to shop, they wouldn't type google or anything else that amazon dot com and they start there. And as a result, if you started to amazon that you're buying from, amazon is kind of innocent until proven guilty. Only if you can't find it there. Do you ever go somewhere else? And it's a massively advantaged position to be in if you can get people to think of amazon anytime you want to buy again. This is 2000,

early two thousands people still thought amazon was a bookstore and that was a very ambitious targets. So that was Plan A and I think they had two plans be Plan B one was what I was working on, which is get really good. If you lose at the starting point for shopping search, you got to be really good at showing up where shopping search happen. So Plan B one was really good at buying ads on google, be more efficient with that than anybody else. So you've got a competitive advantage there and then Plan B two was to go compete with google and amazon people, I don't remember this now, they had its own search engine called A nine, tried to go head to head with google and that, that didn't go very far, it turns out that's really hard. So we made a run at Plan B, one of the stuff I was working at, but ultimately amazon may plan A happened and I think the key strategy for Plan A was prime because once you got somebody in a prime membership, you know, they always want to use it and so they want to start at amazon to see what could they get with free two day shipping before they ever go anywhere else.

And yeah, 27 episodes that talk about how plan plan ended up happening, but you're right, like an episode interviewing Greg Greeley about Prime is probably very high on the list and I already spoke with Dwayne Bowman and Ruben Ortega a little bit about a nine. So there will be an episode on that at some point down the road as well. And so be one what you worked on for paid. I'm assuming there was like a B three as well, which is getting good at organic search. I don't know if that's something you ever tackled or just something that your personalization teams worked on later as

well. My team ultimately on both. So I was considering both of those as variations on B1 B1 to show up well with Google and of course the best way to show up in Google is to be the top of the kind of organic or free results and not have to pay for traffic and then if you don't succeed at that, then then paying for placement is potentially a good idea. In fact, we did a lot of work trying to figure out if you were high on the natural search results. Did it make sense to also buy an ad, you know, to what extent with traffic cannibalized? And that was a very difficult thing to study. Google did not want us to study it for obvious reasons and yet we tried to get

to work on it. Okay, So you gave the background where you think the idea came from and your interest? Let's talk about how did you go about building it? You know, when you said all right, I'm taking over the amazon paid placements on google. Did you approach it as a six week project or did you guys know from the beginning that this was going to be uh, you know, a many months or multi year project to get right?

Remember lobbying matt round and he gave me two software engineers myself included for for three months, which was like in that context, like an incredibly generous gift of resources to be able to do anything. And so the idea was, let's run this for three months, let's see what we can do in three months and then we'll look at the numbers and see whether it's worth doing anything else. So okay, a couple of things strategically were clear from day 11 was the way everyone else's buying ads on google at that point was completely manual. You'd go to google's web uI, you type in the keywords you want to show up for and you'd write your ad and it's a bit on the ads you couldn't do at amazon scale without automation. And the other thing that kind of relates to that is web search is incredibly long tail and incredibly heavy tail. The numbers from the time where I think 300 million unique searches per month. And of course amazon had I think they're single digit millions or low double digit millions products in the catalog at that point. And so everyone who is doing ads, sorry I missed one piece of context here. That's important. Google sold the ads in an auction and this is like one of their brilliant innovations is you pick a keyword you'd write your ad and then you bid how much you're willing to pay for click.

And they would look at what your bid is and then look at what your click through it and your ad is that you multiply those two together. You effectively get what you're paid per time they show your ad for impression. And so they would rank the ads by payment per impression. So you can get your ads to show up higher either by making ones that people are more likely to click on or just being willing to pay more for the clicks. And so you had to know. So it was competitive to show up high because everyone else was doing this manually. They could only go after the most popular keywords. And so if you look at the cross those 300 million searches, the vast majority that had zero ads. you know. But if you search on something really popular late books of course there's a whole stack of ads. And so for our sort of M. V. P. For our version one we said we're going to ignore the popular stuff. We're going to only look at the unpopular stuff and we will buy scads and scads of keywords that are searched on infrequently.

And in fact we don't even have to decide what to bid in the auction. We're going to either bid the minimum bid, we're gonna turn the ad off. And so it would be either five cents or zero. There is the only thing that knew how to do an inversion one and it would pick like large amounts of obscure keywords.

Did you get the search terms just by looking at our logs of what people were searching for on amazon? It's so long ago that google was still passing search information two sites because they stopped doing that

right. It turns out both of those are far better ideas on what we actually did inversion wide. Uh What we did in version one was we like sift through the whole catalog and we look at like the titles of items and authors of what not to try to use those as keywords. And I don't know why we just didn't think of using the search logs. It's a far better idea. And once we had that idea, it worked way better. We launched without that. One of the things that was key to the way we architected the whole thing was we knew there'd be a lot of different strategies for how you can decide how to make ads. And if you were going through the catalog, you potentially know this is the name of a book. For example, is exactly what book it is. And you'd be able to put the price of the book and the ad. And so we have these things that were like kind of mad Libs, but there are different mad Libs depending how you came up with the ad. So there was one that was basically single product ads and there was another one that would look at,

we know about series of books, like the harry potter series. If you know it's a series, you can write the copy of the ads, so it's like a little bit better. And then eventually had the idea that you were just talking about it. Like we could go through the search logs and that was another source of keywords. Another way to build mad Libs and the whole system that we built basically had an internal ap that you could load adds into and then it had this system that would basically track what people would buy or not buy after clicking on the ads, figure out whether it was a good at, and if it was, it would bid it up and if it wasn't just turn it off. And so one thing that's nice about that is that he woke up in the morning, you had a new idea for how to pick keywords and write ads. You could code it up, loaded a bunch of the ads and the system would separate the wheat from the chaff just by tracking what was working well.

What wasn't so two questions you weren't writing the ads were you? I mean you didn't have an editor where the ads being written, you know, via programmatic sentence writing and then were you factoring in things like what was the contribution margin of a product so that you would know what the maximum amount it would be. I don't understand. You started at the five cent level so you don't have to worry about it there. But like you know, did you factor in availability? Hey, this thing is out of stock so we don't, even though it's a great ad, we don't want to be running it. Like how complicated was

it Basically all the things you just mentioned at least for the time it was kind of version 1.5. So if it were the single product ads it would look at availability and it wouldn't buy ads for things that were not in stock. And the way it would kind of figure out was an ad worth buying or not is it would track every click and it would track what people bought after every click. And we had a contribution margin data on a per product basis. This is some of those actually very very difficult for him to create and make high quality but at least we thought we knew how much money we made for many different products sale and so it would add up all those contribution profits and that would tell you what a click was worth. And then it would did kind of a percentage of that click, knowing that it would make some money on the spread and kind of guaranteed not to lose money. So that's the way it would work kind of a mature categories. And for categories that were new at the time, like apparel was new, jewelry, was new. And at that point it made sense to be an investment mode and to spend a higher percentage of sales. So on a product by product line basis, you could tell it do you want to make money or you want to drive sales? And if you want to drive sales, what percentage of revenue are you willing to spend on ads?

And it had had those two modes. Now you were talking about the ad writing and it was basically mad Libs. So you can have these little templates and then you fill in the templates with whatever the title of the book or the name of the series or what the search, whatever the search keyword was. And we don't have any editors and I read a lot of the templates myself. This is very amazon, you know, what are the resources for that? What you're the resources for that. And I just get it done. Don't ask anybody just go make it happen. And I'll tell a little funny story on this. I ultimately owned ECM and Ceo. So showing up again results as well as the paid results. And so I was responsible, among other things of optimizing the amazon homepage to show up in google.

I personally wrote the title tag of the amazon homepage to get the right keywords in there and nearly 20 years later, they haven't changed it, it's still the title I wrote and it reads funny because it's like amazon dot com, online shopping for apparel, computer software. Dvds, it's like Amazon still advertises on their homepage. This sell dvds,

You have to be pretty bold nowadays to go in there and change that though. You're like, is this really a good idea? You know, for some 23 year old head

Is weird because the last 23 year old left in there, you know, two decades

ago and it seems to have worked to get all the data from amazon. Was there an amazon website? Were you leveraging the amazon web service for the commerce data or were you going in getting it directly from databases Without that a p i

e commerce web services approximately didn't exist at that time. They were just getting created. And so there were other kind of pre web services apis that we mostly used. And I think over time we switched to the web services. Api is a lot of that might have actually been after my

time. One other question because you mentioned that you really weren't talking to google. But how did you place the ads if google didn't have an aPI to work with at the time? I'm assuming they do now but

they do now. They didn't then that's a great question. We built a screen scraper and so at the time I think you were allowed to have only 2500 keywords per google advertising account. And so we built this little screen scraper that was basically a virtual web browser. And it would it would go to Adwords. It would open an Adwords account when you put in amazon's credit card to pay for the ads and then it would like put in 2500 keywords. And then when it ran out of space from where keywords it would open another account where keywords and we did that without calling anybody google. And eventually they called us like what

are you doing? Like we're sending you money, you should be happy.

That's right. That's right. That was a whole adventure. Remember at one point I woke up in the morning the system would send emails email reports of how much money it was making and whether anything had gone wrong. Get up in the morning. The email reports say that it can't download any data from google and nothing's working. And so at this point we did kind of have a sales contact, they call them up and see what's going on. They said well your system overnight took down adwords, so we had to lock you out. Okay. Well how did it do that? Well, I woke up at three in the morning and it requested like dozens of expensive to compute reports and parallel and that was enough to crash Edwards eventually down the road a little bit. They said, look, why don't we build you an API. And so they did,

one other thing you mentioned it was you and who was the other engineer that was working on that with you

in the very early days, it was Joanna Power and then Quincy Hot built the front end for some of the reporting tool. So it was really, it was like 2.5 engineers. Quincy only counts as a half because she wasn't a full time.

Did it work? Like did it work out of the gate? Was it a massive overnight success? Because in my mind this thing was a runaway success out of the gate, but maybe that's not

the case, not out of the gate. And so I talked earlier about this notion of the strategy will be targeted the tail, the large number of key words that each one would not show up very often that would target very narrowly to specific products. And so in version one, we kind of overshot on that strategy. And so we picked like a bunch of obscure keywords including this thing was making up keywords that no whatever searched on. But when it hit it hit like spectacularly so that this thing would pay five cents and sell it, sell a book for 20 bucks or Hella Digital, you would have the individual model numbers of digital cameras. And so if you search on the model number and amazon have that one featured, of course, amazon had good prices, very high probability you click on that ad and you buy the camera. And so it was extraordinarily profitable because we're getting sales for five cents that said it was so far on the tail, it wasn't getting very many. And I was so young and naive at the time, I didn't realize that I was supposed to build something big and I was very focused on this idea of like you know, what margins can I achieve with these really cheap five cent clicks And I'm running into Jason Kyler in the parking lot on the way in one day and Jason

was for the listener. Jason Kyler was I think a senior vice president in charge of media pretty much. And now he runs like a digital version of HBO.

But at the time he was Senior Vice president of worldwide consumer software, I think he had run the media business before that. But he moved over his running all of the software at this point. And I think he runs warnermedia now, like all of it. He ran you off for a while. So Jason's been phenomenally successful. And so Jason said well how much stuff have you sold? And I like say with pride, I think we sold 400 units since we watched you. Like what? That's it, Like why are you selling millions of units? And we only spent like $10 to uh to sell those 400 units. But he wasn't very happy about the ratio. He was like why is this not larger? And so I got called into his office along with Jeff Holden who was the VP over our area and had to got told that I didn't have a plan to make this work and I really better come up with a plan.

And so at that point I realized, I still don't know why I was in the hot seat, but I knew I was on the hot seat to make this thing scale up. And so we went through this crunch period where we launched two new features that turned out being a couple of things that made it really big and future. Number one was like David, you had the idea like the very first thing you thought of it, which is just using amazon search logs. Come up with search terms and then just linking those adds to the search results and not very good reasons. We had resisted using that idea earlier. But it turns out is a great source of keywords and the search results, especially with the work that Reuben and Dwayne had done with optimizing the first few search results. The first few were really good. And it turned out people didn't mind. We thought people would mind clicking from one page of search results to another page of search results. But it turned out not to be true, It worked really

well if it's a good page, which has what they want, you know, it's a minor

speed bump. It's like, what's one more click right? Digital Rebel model three. Whatever click on that. And now you're an amazon, click on the same thing, It's not the end of the world, you're on, you're on the product detail page. Pretty quickly. We built that and then we built the, if we called it the variable auto bidder, which was this module that was smart enough to bid numbers other than zero and five cents. So it had some way of figuring what percentage of the gross profit Do I want a bid in order to maximize net profit. And we launched those two things, I think it was like July or August of 2003. And then the thing took off like a rocket

ship by that time did google take you off the credit card? Like you know, I mean we're we still paying because had the relation ship gotten closer with google along the way just because of volumes and api development and everything else? Or was it still

very at that point we were still in the corporate credit card. I remember amazon launched the apparel store,

I think

it was a three. And the search advertising was considered like a big part of the promotion and launch strategy for the apparel store. And so we were kind of on the hot seat with that and that the corporate credit card hits credit limit over the weekend. And so what did I do? I put up my personal credit card. So for a little while, all families on the google advertising was on my personal card and I got a monster expense

report, Must have like nine million credit card miles.

I got a lot of cash back, but it was actually hard to get the reimbursement. So I remember thinking probably my manager can improve this. I'll put it into the director, Your director has approved this, but he doesn't have enough authority. So it's gone to your VP, your VP has approved. It doesn't have enough authority to go through SCP and then end up in the CFOs desk. And ultimately, the only way I could pay that credit card bill that month was to go knock on top school tax door. It's a pretty please. Would you approve my expense

report? That's funny. Tom Szkutak was the CFO at the time at Amazon, so took off like a rocket ship. Do you remember the numbers was a rocket ship, $1 million dollars a you know a month, a week a day and in ad spend or

like I actually have still a chart that shows the sales

ramp. Please email that to me so we can put it in the blog post. So that would be great if you can send that

over. This was the chart my team gave me when I moved on. The system is called Urubamba which is a river that is a tributary to the amazon which is how I got the name. And so those lines, there are O. P. S. Which stands for order product sales. So it's basically sales plotted over time. So you could tell early on we weren't selling very much. And then you see these spikes what were the spikes? That was the holiday shopping season. Once we kind of figured out a couple of the key things then it had some massive growth in sales. So that green line there is the cumulative so it ramped pretty nicely.

That's awesome. If you can shoot a picture of it and send it to me. I'd love to put it in the post itself. That's great. And so side question was this one of the first two pizza teams or was that even established at that time? Because I kind of remember it as such but I don't know if that's the way it was thought

of. Yeah I think it was one of the first two pizza teams. And so the audience doesn't know the concept of two pizza team was a team that's no larger than can be fed by two pizzas. So eight or 10 people tops. I guess it depends on the size of the pizzas and size of the people. But the idea was you would work on a separable vision and you would have something called a fitness function, which is a quantitative way of measuring Are you succeeding at what you're doing? And so this thing naturally lend itself to very simple fitness functions basically became a pl how much profit are you bringing in and how much you spent on ads and subtract those two numbers. And that's roughly how well you're doing with some adjustments in there for things like new stores that we want to promote, where we were willing to be unprofitable.

Were there any other big mistakes made along the way? I don't know if it's obviously a massive, monstrous big win for the company. Were there any other sort of funny mistakes along the way, like rampant, bad bidding or unprofitable bidding that took a couple days

to find and we can probably fill up several podcasts at the mistakes we made, but I'll tell, I'll tell a couple of stories the most cringe worthy moments for me. One was, so we were doing this not just google, but also in yahoo because it turned out back then, yahoo had enough market share that you wanted to work with both of them. And google was very highly automated in basically every piece of how this whole thing worked and yahoo was not. So when you submit an ad to google, it would go through an automated editorial check like, hey, are you buying swear words, just look like it's poor. You know, it did, it would be flagged for an editor but they're kind of this automated approval. Most of the jobs we get out.

Yeah who would have a human look at every single one and here we are in amazon, we're just throwing millions and millions adds at yahoo. I think yahoo had more people working on manually reviewing our ads that we had people working on the whole team that built the system that created the ads by like a factor of two or three. We were creating a massive employment at yahoo with this system and so eventually they create article and they were like, well we trust amazon so we'll just turn off all the editorial checks and we're just like ought to approve all of your ads. It was very nice that they were so trusting but it was actually very poor decision on their part because we didn't have any automated checks on our side. This thing would buy ads for everything. And so there was a period of time where we were the only advertiser on yahoo for the keyword por they weren't supposed to have ads on porn, but because we got around the editorial checks, we could buy the keyword porn. And I mentioned earlier that one of the strategies for coming up with these mad lived adds was to look for book titles. So there's a book called 688 Ways to Make Love like a porn star or something like that. You know, I think it was actually how to make love like a porn star. That was the title by Jenna Jameson. And so the headline says how to make love like a porn star below that,

it says you know, free super saver shipping on orders over 25 bucks. But the headline system make love like a porn star. So if you go to yahoo and you're someone who's searching on porn and you see how to make love like a porn star, like you click on it but it turns out a book is not at all what you're looking for in that context and at the time the part of the system that was automatically looking to see who had clicked on what and what was selling and what wasn't and turning off the loser ads was broken and so and it was, it was down for some ungodly amount of time, like five or six days and the time he was just horrific enough that the system bought this ad on yahoo for porn and didn't discover five days that it sold, not generally like hundreds of thousands of clicks and sold literally zero items, literally zero. And so I remember having to go into my boss's office at that point and explain why it is that we had bought a bunch of completely worthless as for porn. So that was one. Uh

I'm assuming that probably led to some type of automation to check very unprofitable ads or something like that. Like ads where there was spending with no purchases and that sort of thing.

Well we have that automation but it was behind and broken so that the whole um you might imagine that there was some very clear way, you could tell who would click through the site and what they had purchased. But it turns out we didn't have that system and the only way we could get through it was going through the web logs from all of the website. And so this this was an enormous amount of data and you had to process it serially because you have to see when someone clicked through from google what the tracking idea was on that you have to follow that user all the way through and see what they purchased. And so it was a lot of data was compute intensive. We didn't have any kind of good infrastructure for distributed compute, so this is all running a small number of compute boxes, kind of single, single threaded and it's something broke. It would get behind and it was really difficult to get it to catch up and you could see the profitability of the system longer you went without closing the loop on what was selling and what wasn't like the worst the financial performance of the whole thing was. And so it turned out that the biggest way to make this thing work better. It was just like speed up that process, Just get better and knowing what people have bought after a click and then turn it back around into a decision to buy the ad or not. But when it broke it was really painful. Which reminds me of another story, I think it was holiday 0405. When you go through the holiday season,

the profitability of everything ramps and so people propensity to buy, you know, if you look at like sales for click during the holiday shopping season is much higher than it is the rest of the year and so you want to bid up your ads and pay more and then as soon as you hit the kind of holiday shipping cut off and people are going to buy as much anymore, you want to rapidly turn the bids back down. And so if you don't build the software for that, you'll have the effect where basically you under pay during the holiday shopping season and you over pay after the holiday shopping season. And so we built some what we thought was a very clever software that would rapidly figure out how was the site wide conversion going and use that to adjust the scores of individual ads. And we thought we were all very clever. But it turns out after that there's a confluence of two bugs. One bug caused us not to turn on new ads when we wanted to and the other bug caused us not to turn off ads who wanted to turn off and so yes, this is a bad combination of bugs. So after the peak of the shopping season, when this thing was like in the black and it was making loads of money all of a sudden it's massively unprofitable and you know, for a couple of days, like I guess the algorithms are taking longer to catch up than we thought they would, but it keeps losing money and really what is going on?

And eventually we figured out that these two bugs existed that caused us basically to only run bad ads, literally the opposite of the set of ads that you want to run what you're actually running. I remember getting a hold it in front of the VPs for that one. And, and I think it was Jeff, hope you said to me this thing was a rocket car. Then the wheels came off and I was like, what? Jeff does not just make it a rocket? Like you're missing the point. Like I remember my performance review that year had a picture of a rocket, carbon. It not necessarily in a great way.

Well look, I wanted to talk actually about organic search but we might have to try to beg some time for a future episode because we're already at 50 minutes. I wanted to ask a little bit about learnings from this like sort of if you step back from it and say you know for future entrepreneurs, what types of things do you think that they should learn from this? And then I do want to spend a couple minutes talking about your post amazon experience because it's really unique as you know. So so when you think about learnings or the types of leadership principles that most resonated with you and in this project what type of things do you think entrepreneurs should be thinking about? Or people innovators inside Cos I mean you were innovator inside amazon that ended up having a multibillion

dollar impact. I mean I was very lucky to have that, there's very lucky to have that so early in my career. So I'll throw out a few random lessons. One is if you can find things at the intersection, if you're a big company and you can find things intersection of stuff you think it's really cool that you'd love to work on and the company thinks it's really valuable, you're going to get some insane acceleration in your career. I totally lucked into that. But I thought the google ads thing was cool and for reasons I had no clue about at the time. This is something I have to update the Ceo on like every three months. And so when I was screwing up I had some really, really smart people telling me I was screwing up and riding my butt to get it fixed quickly. And so it was this accelerated aging process. I got to learn way faster. They would have another context. I think that's one lesson. The other thing that made this fun,

interesting quick learning is it was very measurable. You could tell end of every day whether you made or lost money buying these ads and at what volume and so you could tell quickly what was working, what wasn't working. And that that led to this process where you can iterate and improve it very clearly. So it's a very, very short gratification cycle. Not everything is like that. If I look through my career, I've learned to live with longer and longer gratification cycles on what I'm doing. But this is fun because it was very short gratification.

Yeah. I mean literally some people would work six months on a project or three months on a project and launch it. It might fail whereas you were able to, once you got it live every single day, even the 400 units sold,

the 400

very highly profitable units sold at least it was, you knew, and once it did get the visibility, you were able to ramp it up very quickly and continue to see.

So you were saying very quickly, but the reality was it took the better part of a year to figure that out. And there was a period time or amazon wasn't sure we should stick with the in house technology and they were competing against some third parties that they thought were smarter. So which is very, very painful. And it was learning for me not to have not to react to that defensively and politically, but how to how to wear the company had not the

team hat. Yeah. So just make sure we get out of here on time and respect your time on. It's a sunday night that we're recording this. So tell me about your post amazon experience and really I want to talk most about boom, but you know maybe just give the short

version and I left amazon and yeah, no six and I was part of a couple different startups. One was founded by Jeff Holden, who I mentioned earlier, it was one of the very first mobile apps companies pre iphone and after that I started a mobile e commerce company called Chema that was required by group on. But what I learned almost in a way kind of bridging back to the stuff we were just talking about. If you really want to work on stuff you're passionate about and the stuff I did it my own company even though we managed to sell it to group on, like I wasn't that excited about and there is nothing like working on internet coupons to make you learn to work on something that you really love that's going to matter to the world. And then I made a left turn in my career. That is very much a non sequitur from my resume to found boom supersonic, which is building high speed passenger airplanes. And my backstory was repeat that

again for everybody. You were personalization engineer and had a couple startups and then you're now building supersonic

planes. Yes, it sounds a little surreal when you put it that way. But no, we want to build you know, I think I had a throwaway line earlier that a lot of things in my career about to making things faster and so we're going to make an air travel faster now. And we've built and rolled out the first independently developed supersonic jet ever. It's going to be flying and not that many months and will ultimately allow tens of millions of people per year to travel and half the time it takes today over long distances.

That's, I'm little goose bunks like I didn't, I didn't actually know it was so close, I've obviously followed along and so did the idea come from early interest in the concord or was it more of a speed? Things like where did you just take that one? Maybe that one side story is it's such a massive vision. Where did that seeds get

planted? I've loved airplanes since I was a kid. My parents will tell you that I've loved airplanes since I was six months old, been flying for fun since I was in college and in my mid twenties when I was in that was worth that startup called Peligro that Jeff Holden had founded. I'm at one point, I was waiting for my girlfriend who was one of these like horrifically delayed flights and I'm just sitting there waiting for thinking like How much better life would be if she arrived at Mach two instead of on the airplanes that existed today. I started researching it and I put a Google alert, supersonic jet because I want to be first to know when I can go buy a ticket and break the sound barrier. And for the better part of a decade, it was just cricket. There's no credible effort to pick up where concord left off and to build something mainstream. And I could

see from

day one for me, this was a very, very human thing. My kids had a grandfather who lived in Hong kong, sadly he passed recently, and the kids got to see a maybe maybe five or six times in their lives period because the time to Hong kong is so long and when I contrast that my own upbringing, I've been an hour and a half away from my grandfather was a kid and I learned so many things from him, like there's no, I'd be an entrepreneur if I work for the relationship I have with my grandfather and I look at this and like why, why do my kids not know their grandpa is because the airplanes that we have today are no fashion, the ones we have when my parents were growing up and it was very, very motivated having spent time in amazon and tech. I grew up in a world where everything is getting faster and better, why aren't airplanes getting faster and better? And so I knew I want another start up and I wanted to work on one where I would never get up in the morning and think why am I doing this? Is it worth the pain? And as entrepreneurs,

if you're driven, which every entrepreneur is, you work at your personal red line and work at your personal red line kind of feels the same no matter what you're doing, you can't go any further than that and you're up against the most you can possibly do. And if it ever gets easier, you just find a way to do more. But it's not the same between every startup is how motivated you are and how worth it is it to kind of run at your red line and go through all those difficult patches that you're variably going to have. And so I thought I would sort of my startup ideas by how happy I would be if it worked and I figured I would cross off the first X number ideas as impossible or bad. But as luck has still working at number one and I got, I got kind of two weeks into the research On it and discovered that most of the conventional wisdom about why supersonic wasn't happening, was wrong or obsolete. And it didn't stand up to very simple quantitative analysis that you could do with basically no background with spreadsheets, you could build out of data on Wikipedia. That was 2014. I spent about a year learning aerospace, reading textbooks like an airplane design.

Class, networking hired my first people in 2015 and today is nearly 200 people. We've rolled out the XP1 demonstrator airplane, which is like I said, it's a ground test now and within the next year it's going to be flying and then our overture passenger airliner, which will take you across oceans and half the time that takes today's airplanes. It's going to be in the air in five years.

It's incredible. Was this at all inspired by what Elon musk and Jeff Bezos are doing

with rockets? Oh yeah. Like there's no way I would have had the courage to do it. I remember sitting in my basement in 2014 thinking about shot up ideas. I think Ellen went from paypal to SpaceX and rockets are probably harder than airplanes. So if he could do that well, you know, at least this is theoretically possible. You know, I know if I said I could do it too, it was uh, you know, I had to sort of decide not to care what I thought I could do and say like, the only way to know what you can do is just to try and so forget about the weather, put all your attention on how and if you fail at it because it was impossible or you weren't up to it at least you know you tried You never have to be like 80 years old on your death bed thinking I wonder what would happen if I tried

to build supersonic jets.

So you can, when you get that mindset you can't lose either it works or you tried.

It's back to that quote. We've all heard many times about Jeff when he was leaving D. E. Shaw and he said and he left in the middle of the year and didn't get his bonus. And he's like I know for certain when I'm 85 and dying that I'm not gonna regret having missed out on that bonus. You know what I mean? But I'll probably regret if I don't take a chance on this this internet thing and it's very resonant.

So honestly, I might not have made that decision if it weren't for that very specific story from Jeff which has really influenced time, make some of the most important decisions in my life.

I love it. Well look, thank you so much for being a guest. I do hope you consider coming back and doing the S. E. O. Version of this with you and or you know somebody else from that team. Because I think that's another really fascinating story because that work is probably also driving billions of dollars in sales. So just the automated advertising story would have been great. But you also shared many other great stories and even the final mentioned, the boom inspiration is great for other entrepreneurs and innovators to here and it's just great to see you. I haven't seen you don't we've spoken but haven't seen you in a in a long time for the audience. Thank you for listening to the Invent like an owner podcast. If you'd like more details about what we discussed today. For one a semi edits and and improvements, please visit Invent like an owner dot com to sign up for my newsletter. I'll also put up any photos from blake for that chart and or photos or videos you have for boom planes. I'd love to include them in the post. So, yeah, Thanks again.

Thanks to. This is a real blast. Thanks for taking me down memory lane and getting to tell some fun stories.

Awesome. So that's it for today and remember. No sniveling. Okay. Mhm. Yeah. Mhm.