watch the log files very closely. It actually was a beautiful thing in my mind because we're taking customer input and using it to make customer experience better every single day, you don't get much better than that. This episode is sponsored by my perfect color dot com. Founded by ex Amazonian Jason shaw. My Perfect color is a paint manufacturer specializing in exact match paint color solutions for touch up paint and marketing applications for businesses. Color obsessed companies such as Yeti coolers, trek bicycles, video Tiffany Jewellers and more rely on my perfect color to match their colors perfectly every time, whether you need paint to develop prototypes, displays, signs, exhibits or to touch up product that has been scratched during fabrication, transit or installation. My perfect color can help to learn more, visit my perfect color dot com. Hello,

I'm dave chappelle and I'd like to welcome you to the event like an owner podcast where I have talked with the amazonians who helped build amazon dot com into one of the world's most valuable companies. This weekly podcast is for entrepreneurs, future business leaders and students of history and not to mention people who want to get a job at amazon. Still, the goal of the podcast is to capture the amazon creation stories and create a historical archive. On that note, I guess are recalling history as best they can. It's possible. Some of the details are fuzzy or wrong. If that's the case, let us know, you can, you know, do it in the comments and email me and I invite future guests or commenters, help us get the facts as straight as they can be. Uh now on with the show today,

I'm thrilled to be talking with Dwayne Bowman and Ruben Ortega who hold a special place in amazon lore as the creators of bottega box or bottega, I call it to take a box, but you'll learn more about during today's episode. They were both quite early Amazon software engineers, starting in 97 is Dwayne in 98 Ruben. So I'm sure they'll both have a lot of interesting stories to share about early website innovations, especially around search, algorithm development and more welcome. Dwayne and Ruben.

Thank you.

Sure.

So let's jump right in.

Can one of you tell the story of bottega or bottega box and you know, maybe put it in perspective is what the experience was like at the time where it was created.

Sounds Good. So I joined Amazon in January of 1998 and we launched the baton to box in March of 1980. So it was three months to I. D eight implement and launch patent and launch. So what was happening on the site at the time is we had one sort order on amazon dot com, It was alphabetical. And so if somebody came to amazon dot com and search for the partner, the results, you would get back. The top result would be

uh, the partner being a john Grisham book or or something like that kind

Of part of being the book by John Grisham. The top result would be 101 date ideas for you and your partner. The 40th result would be how to make partner in your organization and you have to click two more pages before you got to the partner by John Grisham and we knew we were going to change the search engine. We had plans to change the search engine, but it's going to take four or five months to re implement what we needed to do. And this was a horrible experience on the site. And we officially had no time to work on this. We had to fix the bugs. We had to scale the site. We had this new search engine that was coming, there was no time to work on it. So over a weekend, what we did was I was looking at the problem and I was looking at the log files and the Eureka moment habit. The Eureka moment was what if we could take all that energy our customers were doing to find the thing that we're looking for and what we ended up doing is we end up mapping the words people used to the things they clicked on and waiting it by the items they bought. We built small index, put it, did a quick little look up algorithm on it using some code that Greg linden implemented. And

we had,

we had a stunning result that happened basically. We had inadvertently discovered the vocabulary of purchase so customers were telling us and doing putting all this effort findings things such that we could build a small sub index and return results that led immediately to purchase. Not only was this monetarily good for amazon, it made the customer experience beautiful. You could come into the search, you know, after a couple of customers typed in the partner one day, the very next day if you typed in the same phrase rather than page down through pages of results, it appeared right at the top of the page. It was kind of amazing and the thing that I really loved about it was that even the data bugs were powerful because if enough customers type owed the same name like limp biscuit, then we suddenly start getting that result at the top of the page because eventually if the title in biscuit and it got and bought this item, they would get they would get that result. Another example was searching Yes,

So it's sort of like early spell check or basically there's no human needed. It's learning that hey, people are typing variations of the term and they're still ending up clicking and buying the resultant product.

Exactly. Or even words that had nothing to do with the ad itself. For example, people would type in Oprah, Empire Falls. The word Oprah has nothing to do with the Book of Empire Falls, except it happened to be Book of the Month Club. What that meant was later on that month or later on the next day, somebody could just come into the website and type in Oprah and they would start getting a relevant set of results for what was going on. And so it was really one of those moments where you, where you build something and look at it and just go, wow, the results were so much better. Everything just was, was so much better after with search results with respect to the relevancy to the customer and the quality of the product.

I was just gonna say Ruben the key phrase you mentioned there and when you're talking about it is, I was looking to the log files one weekend. This is something that log files or something that you don't particularly read on at

your leisure time, right, sort of just like, uh,

you know, it's basically a record of every click on the site is what the log files is. So the thing about Reuben and I both of us is we spent tons of time sort of gripping to the log files and seeing what the problems were with the issues where how can we make the site better? And this is one instance where is an amazing sort of leap and this is 98 so this is actually pre google, you know, this is a long time ago. But yeah, it's even better than actually trying to use Belichick with a dictionary because many of the word you're typing in or not in the dictionary, their proper names, et cetera. So we can actually handle all these misspellings just from looking at data, the log files and building these indexes. It is ironic you use the example of the partner and probably you guys were both looking through log files because you should have been finding a partner

at the time. So so

I again I remember it distinctly but you mentioned that you leverage some work that Greg linden had been doing is basically when people would see is instead of having to go to page five or whatever, they would see the top three results and you shared that screen grab with me that will put in the post was that used in other places on the website or detail pages for instance, did they use different things for, you know, people who bought this? Also bought these other types of things or was it all working on the same, you know, core set of

solutions? It was the start of a set of solutions because when we discovered that if you can use words to map to behaviors and waited by the purchase, it really began a whole fan out of innovation that came out that came out of that. And so we could start, we start doing things like words. People use to buy products, other words, people use to buy products, reorganize browse trees so that if customers would frequently go to the book site and navigate the browse treat of bestseller, science fiction, hardcore science fiction and then click on game of Thrones. We could reorganize the browse trees such that it would lift those relevant categories up, reducing the amount of energy would take another customer to find things. I know. Eventually personalization ended up using words As weightings as part of their detail page results. But it took a while to get there because it was, there was a lot of work to do in 1988.

This is a nontechnical person asking the question, but why was it that alphabetical was the only option at the time? Like every engineer whoever built the first search engine would say we've got to be able to bring back results in a different way. So were there other simple algorithms to display search results and they just didn't work very well or had someone never really thought about a purchase experience or

even a search experience

right? About how to get the best result. Not the one that starts with the

letter A this

we're going to Dwayne Yeah, I don't think that a lot of thought was given to it back in 96 97 when the site was just spinning up, I mean shell, you know, it was doing everything on the site and he sort of thought about as a database problems. So and the way people searched back then pre google this kind of application was You had fields of data, you have the author field, you have title field, you have date fields and you want to put terms into those fields and see if you can get the right term set to get the result you want. Right? That's what the interface looks like. Actually when I got there in 97 there wasn't a single box. You just type of word in like like a Google box, right? That wasn't existing back then. It was an altar title date search basically. So yes,

it's clear now that you don't want that kind of thing on amazon.com site and relevance wasn't something that was easily gotten sort of arrived in that kind of situation. Right? So we did look at changing the search engine pretty quickly. We got there and I guess later 98 or 99 cultural change the search engine. But we have to live with what we had at that point. And this is the best way we can go. Are you saying that reminds me my local library, I still get books at the library because I'm frugal, eternally frugal. And their search interface still looks like, you know, the

actually google still

has one too where you can expand to search on multiple variable field. So so I read something that Reuben had written that it was a big jump, like what was the sales jump from bottega when it launched? And the secondary question on that is did we even have a B labs, you know, web labs or a B test at the time? Or did you literally just say this is better? We're gonna slap it up and we'll look at the change in

sales volume. Well, it was double digit percentages attributable sales. You can always argue those customers would have rewritten their query. Eventually, somebody had gotten the partners not seeing the result in typed in the partner, john Grisham and eventually they could work their way to it. But it was really more of just making it easier for customers to buy things. And so there was no extra extra revenue other than the assumption was it made it easier by once they would look for more products and it would make it easy to buy more products.

There was no A B testing set up at that point, basically we just tried it. Uh, it was clear the experience was much better. So we put it in production and watch the log files very carefully for 24 hours to make sure nothing was fine. But the experience was so much better. Even the sales numbers trade bill didn't really matter that much. It was just a clear experience. It is funny, it's a way of thinking about measuring the right thing to. So for instance, you probably really negatively impacted page views on the site, you know, and time on site, you know, because I remember getting there and people talking about that as an example, Ebay people spend tons of time there.

Whereas amazon's heavy customers did not spend a lot of time because you and people like you and teams got really good at helping them quickly find what they were looking for. So they, you know, get in, get out and hopefully have a good experience. So longer term a change like this. How did it ripple through? I mean you mentioned spell correct or you know, an early version of it. How do things like this ripple through the little bouncing around a little bit? Because when you were first hired Ruben, you weren't hired to solve this problem, right? You were hired to improve search. Let's go back to that. Like was that your first task or Dwayne you hired Reuben,

is that correct? Yes. So like was that his first task was like, hey Ruben, your job is to fix the search for you and this team is gonna fix search. And how was that prioritized? Because this is obviously made sense to do before fixing search because it was sort of a smart hack to fix search until you know. Yeah, Right. Yeah, I don't remember exactly the priority list the exact time we hired Ruben, you know, I, I had been hired spring of 97, so I've just been there a year basically when Ruben came along and the search team was basically just me. I think Ruben was the next guy.

I think eventually we hired several more people and we had had the catalogue group as well working with them. But yeah, I think the bottega thing was not on anybody's priority list. It was a weekend hack project that turned into something much bigger and uh rubens, right, has fallen all kinds of related, you know, sort of technologies down the line that became experience, became quite a bit better related searches. I just was reminded that when I saw the screenshot there, the spell checking thing. So yeah, I remember the priority list, but I think the big priority was migrating to the new search engine, which actually was just another stop gap because a couple years later we actually wrote a whole another search engine in the house and migrated that

as well. The we replaced our internal search engine with alta vista and we had learned so much from the bottom of the box experience and from the data experience the day we launched, it was irrelevant because we got better returning relevant results than an off the shelf search engine. The technology of the day was very much focused on the relevancy of the day was very much focused on, Can I analyze this document better and get the right result for a customer? For a person, Right. And what we discovered was that we needed an index that was capable of being updated hourly daily and being able to wait off metrics that were outside of the context of a simple document we originally were swapping with. The virtue that stopgap engine was that it was capable of handling international languages. We knew that we were heading into Germany and France soon and we needed searching that was capable of handling multiple languages, but we didn't have the team people building that

yet. And the U. K. With all those extra use, you know, and everything. And so we're search results at that point person specific. And what I mean is did they factor in the things that Dwayne had searched for over the past year? Or was it still just looking at the word that was typed into the box? And it was just a word against the, the index. That's, that's all we had at

that point because somebody searches for partner.

You actually don't have any idea if they're looking for john Grisham becoming a partner or finding a partner. Right. And so and so that was just still a limitation of this basically. Like how would you wait in that example, if you can remember if someone just searched partner, how would you, how would the team, the search team think about prioritizing each of those results, would you say? Look john somebody searches for the word partner, john Grisham's in that in top three? Or did it change based on, you know, slight variations like with him without the etcetera.

One of the innovations that came in with that was that we decayed the data over windows of time. And so this was actually the failure case of Barnes and Noble at the time I was comparing our results to Barnes Noble because they actually had more than one sort order and even they were failing on capturing relevant results in a timely fashion. You can do a search for the word Cat and their top result was capturing the rye. One has nothing to do with the other and I'm sure capturing the rise the best selling book of all time. But somebody using the word cat is not looking for

capturing the ride. Yeah. When you look at the site now assuming it's massively better in your time. There were there big strides made and things like that even or was it still so early that there was low hanging fruit for things like improving personalized search and that sort of thing. Did

I mention there was no time.

Yeah, I've heard that. That's that's

been a constant theme.

Yeah. There was a lot of low hanging fruit back in 97. For sure. And we were just, you know, the big thing in 97 was even we're just books alone in print out of print books. It's always to worry about. But we had to worry about getting a catalog up every day. Pretty much if new book comes out, you want the book to be in the catalog and be on sale that day comes out right? You don't want people waiting. So just the and I don't remember all the details of this. But just the onerous process of taking in input feeds from baker and taylor and the publishers and building a catalog that was accurate and had the right information in it and putting on the site just that was herculean at that time trying to get it done. And so that was a lot of our time just figuring that part out. And then to search things tunnel of hanging fruit because we hadn't been thought about really Before 97. 1 thing that revenue mentioned and maybe I didn't understand it. So I'll go back to it. You mentioned swapping out from the original Amazon search engine that I'm assuming shell and team built to Alta Vista. And did you say that that didn't really work or it was only good for a short period of time before it had to be improved again. Like was that considered off the shelf at that time? Was that the best that existed or how was that

decision made? Basically. It was, it was a piece of off the shelf software that supported multiple languages and that was the key constraint and multiple sort orders. So that is, it was a key bit of infrastructure that did it. And it was not a skill that we had in the house. When Dwayne hired me, I had zero search engine experience. I looked at the code as written and was able to tweak it to make it go better and faster and stronger. But it wasn't until we hired more people on the team that actually had search engine experience that made the product of the thing that it is today and bottega no longer exists on the side. I don't know the day it was removed, but I know it was past 2000 and two at least. But I know that what is there now is, you know, science fiction compared to what we started with.

When you think back to somewhere again, it was probably in your article, you talked about the scaling of search both probably in terms of just sheer volume of searches happening every minute, right? To probably how complicated they were in terms of what data is they're going to look at. Maybe this fell more on Dwayne. Like how did you balance again, new features versus scaling and supporting and making sure it was fast and you know, stable if that's the right word. Yeah, I don't remember the numbers maybe Ruben members. The numbers better than I do in terms of how many searches per minute. There was a ton of searching going on on the side and it and the scaling just keeping up with the scaling was an issue for sure. And uh you know, kim talked about in your podcast with her the migration from a single server to multiple servers and all that. And putting indexes on each server, putting the catalog on each server.

That was a big piece of scaling. But just to get just to keep up with volume was a big deal. You know, the ultimate solution was a good off the shelf. It was probably state of the art in 97 98. You know, we pretty much wrote something a year after that or two years after that. It was better in house basically for our needs. But yeah, it seemed like every day was mostly consumed with just not breaking the site and keeping the scaling going in the early days. Was it more just throwing hardware at it or was it a mix of both? It was a mix. Yeah, yeah. It was definitely a mix. When you write something as complicated as the amazon back in with basically one person writing it, you know, for the first year or so. There's tons of low hanging fruit you can do to dr mayes and so we spend a lot of time doing

that. The big win in migration was going to multiple servers. Success has many parents and everybody claims to have, you know, you know, started web services at amazon. But I know that I started in amazon because the very first web service at amazon was when we decided to take the search off the same box as the website, put it on its own fleet of servers and it was using xml over http to talk. And the name of our first search server was overdose. We had no experience in the house and how to deploy multiple binaries. So we would take that same front end server, send it to the back end. We added a little tag that says ampersand, x m l equals true if that came back. And we need to send, send back a set of xml results.

And what if it didn't come back? What are some examples that would make it break or would you have to re request or how would

that work? Failure wasn't an option. That was an option. So it always so

it always returns through.

Well, actually that was one of them. So to call to another friend and colleague there the day we did that deployment every all the friend and servers at the time immediately went from a load of 60% utilization down to 30% utilization. And I got a phone call from jesse Robin saying you broke the site and I said, what do you mean? I broke the site. He said, we've just lost half the competent compute on the site, and we looked at it and everything was working, and in fact, the site was working better, and what had happened was what we didn't know until we had done this change was that there was so much memory competition on the servers from the search engine operating at the same time as the website software, that just by removing both, both started running more efficiently, because you had a machine that was dedicated to running just the search service, and you had a machine that dedicated running just website and they weren't competing for the same resources and the same memory,

you mentioned, Jesse, can you give a quick explanation of what his role was? And then also was his conclusion, you know, based on measuring the wrong thing or because you can also could have just said, hey, jesse, our sales off, you know, our shopping carts off or you know, so what was Jessie's role at the company at the time,

jesse Robbins is the master of disaster. He was the one who was owned the infrastructure for amazon at the time and he was the one who was dealing with a lot of the operational lagu and so similar to us looking at log files, the only tools he had, we're looking at loading capacity on a fleet of servers and then trying to figure out what happened. I'm positive that the instrumentation we have on the site, it will better correlate revenue to activity. But again, at that time, all we can do is call each other up and say, what did you just change and try to figure it out?

We spent a lot of time talking about Bottega, rightly so, but when you step back and squint and stop think about Amazon you think about entrepreneurs, who you probably meet with regularly. What are the lessons from this? Not an Amazon 1997 perspective, but in a under resourced, you know, massively high growth situation, what do you think the lessons are generalized to take out of this for future entrepreneurs? I just think, you know, bias for action. A lot of us who you hire of course hiring is the biggest thing. And so if you get to hire people like Ruben who can come in and spend most of their day thinking about the problems and you know, great ways to solve the problems, that's the key right there and someone who can actually not sit around and you know,

designed for six months, the perfect solution the problem can they actually take what they have, you know, Berkeley DBS and the code they have at their, at their fingertips and cobble something together that actually works and then perfected as time goes on to me, that's the biggest thing. I mean, when I think about it, I think about it, like I've worked with a lot of engineers and some focus on the, you know, the complexity of the solution. I look at bottega and it's just like you're actually looking at it through a lens of what's the customer problem here and how do we quickly solve it with an elegant solution that might not be perfect, you know, but in this case it was nearly perfect, you know,

given the state of things. But that's what jumped out at me Because it lasted for, I told the side story and we'll get to the naming. I was there, I started in May of 98, I had no idea of Bottega had anything to do with you. Like it was

just, it wasn't the thing,

it was the

thing atop the search

results. That was awesome, you know, and where you spend all your time. So I also think about just the data, like because you didn't have a B testing at the time, you sort of knew what the result had to be and it was assuming watched very, very closely when the change was made. We did, we watch it very closely, watched the log files very closely. It actually was a beautiful thing in my mind because we're taking customer input and using it to make customer experience better every single day. To me that you don't get much better than that, right? So you're using your customers using the volume of stuff we have on the site to make the experience so much better. And so I sort of just, I was amazed at how quickly Ruben got it working and sort amazed at how well it worked. And the following things that enabled as

well. This was one of those cases where you first think the team for putting it all together because the first thing that came out is that I learned from is that signal is more important than analysis that by signal is so strong and that by signal wouldn't have been there had we not had people, except we had we not been accepting credit card for purchases, had we not had a website, had we not had a product that people want to use in the first place. And so you know that your regarding your startup advice, if you have a strong signal that will take up the place of a lot of sophisticated analysis. In fact, it's better than sophisticated analysis. Then the other thing about the team is Dwayne, I know protected me working on this, I was tardy chatting about this topic a few years ago with Joel Spiegel and he had mentioned he had just gone to Jeff Bezos with the plan of what, you know, we're going to do with search catalog for the year we launched and then he had to go back to Jeff and say I made a mistake and Dwayne had a kind of buffer the amount of time I spent on this versus the other things because this was fun. This is, this is, you know, this is, this was enjoyable. I mean your head must

have been blowing up when you're working on that over the weekend and you're looking at the results and you're comparing this screen from that screen, you're like, oh my God, people are gonna freak out. You know, you guys must have been so excited. We were. And I imagine you showing it to the first groups of engineers and people must have just been smiling like it was so much better. Yeah, I mean I clearly remember, I don't remember much about it, but I remember clearly, you know, whiteboard sessions with Ruben in my office and just feeling a whole whiteboard full of this stuff and then him going off and implementing it in a matter of hours, you know, so that's, it's a great feeling we didn't really mention it explicitly. So maybe just tell the fun trivia how bottega got its name and sort of who's behind it.

So what happened was the person working on the front end of the time was then a software developer by the name of Alexa Edelman. And he kept calling Ruben Dwayne results Ruben Dwayne results And there's a bias in the organization not to name things after people. And so we're just like, you know, you know, not that country, you know, it's a similar search. And then he eventually flipped the name and put in the secret code, the code. He used to invoke it where he ended up flipping it to be bottega. And it was so innocuous that it was like, well I guess we can let that one go and, and so that is how it got its name. Uh,

that's pretty cool. Like I said,

I didn't know. So I think

it fulfilled the innocuous part of its submission there because I was there for months before I had any connection to the two of you for better or for worse. But I think you guys would think for for better

jumping

around a little bit. You mentioned about the team. So Dwayne, you recruited Ruben.

Can you just talk about how many

people were There when you got there, Dwayne? And did bar raisers exist? We haven't really talked about bar raisers or did that existed? You guys helped create it? Like what was it like in 97 and 98? I think when I got there, I got there right before kim Rock Miller probably. And so in her words, there was about 35 people doing everything, website, front end, back end everything. So that was the way I remember it basically, you know, the second floor or whatever that was in the second Avenue building. And I don't know that the word bar raiser,

you know, the bar raiser program was not there. Certainly. Even when I left, I don't think that was, was there officially? But the idea of it was there right away. I remember being sort of surprised by how we did our whole interview loop and post interview meetings and the whole thing. So we were very focused on getting the right people on the bus, which is smart and it came right from Jeff and and the people he hired, I think Joel did a great job of recruiting. He recruit a bunch of people from his past, you know, so I think more than anything else, that was probably a big reason for why amazon could scale so quickly. Do you have any inputs on that ribbon?

Like thinking about but not only being recruited, but sort of becoming a good interviewer. When I talked with startups, you know, sort of an advisory roles. I think the thing that comes up more than anything is leadership principles and baking that into the hiring process and performance process and sort of knowing what you're looking for and then trying to suss that out in the interview. I don't know Reuben, I think you talk with a lot of startups now. Like how do you think about it back to then

and now? So there's like three questions. There was a piece of apart what I was recruited in by Erica Lock who introduced me to, to Dwayne and to Joel and a bunch of my friends were working there and I thought I can work there for six months. Let's see, let's see how this works out, ended up spending nine years. The bar raisers itself really emphasized the importance of being team. I was one of the first bar raisers I was with that first group including and I think Andrew certain started it. He was part of that initial courtroom and carried the bar raiser flag for quite a while and we were pulled into a room and they said you have been identified as some of our strongest interviewers. And the goal was the bar raisers was not only to make sure that we had impartial interview loops but to make sure that the process of the interview went well just to make sure the managers were trying to hire people in desperation or because we needed a lot of people immediately. And just to make sure that that you know we were having a debrief, making sure that just you know, similar to I listen to kim rick miller story earlier and there wasn't a lot of process at the time, but you have to put your name on something, you have to put your name and said this group was interviewed. This candidate was interviewed by the following group and who is your bar raiser.

And that was effectively almost like the code review. Did you have this person? So if that candidate came in and there was an end up being a immediate poor match, we had some accountability for it. And so now in talking with new start ups, a lot of it is really a reflection of the same, you know, who's your team and how do they find new people on your team? Referral networks are good, but referral networks need checks and referral networks are inherently biased. We're biased by the schools, we go to our advised by the organizations we hang out in. And so if having somebody on a loop dedicated to mitigating that bias is really

helpful. I found recruiting and interviewing very stressful, you know, because admittedly I wasn't good at it in the beginning. And so that was like a performance goal one year to literally just get good at interviewing. But yeah, it's that whole idea of, I've got to make a vote public before, I know how everybody else felt and I need to document it and you know, have a basis for it. I'll do a whole episode on this with someone and because it really is one of the biggest things I took from it, but the natural outcome of it from what we're talking about and kim and Joel and it's just the quality of the people that amazon especially early. I mean it's obviously just grown from there by repeating the process, but just incredible people in those early groups, recruiting other incredible people, you know, so Dwayne allowing yourself to scale Ruben,

etcetera. So, and I think Dave are you mentioned before that hiring engineers at amazon in general was a tough go at first because amazon dot com wasn't a sexy engineering problem necessarily. It was a website, You know, it didn't seem that cool or problematic or whatever difficult, But I would say the exception of that was probably the search team because the search problem was very interesting and complex problem that people want to dig into when they saw what the volume was going to the site. So once we got past 98 or so hiring a great search engineers wasn't that difficult. They wanted to come to us. They wanted to get their teeth into someplace that's just handling volume we were handling and problems that we're trying to service the right thing for the customer at the right time. All that, all that stuff. So that's the exception. I think I could see how someone who's interested in search would quickly grok the, you know, we have to figure out all these product attributes and we have to know how to present it and personalize it.

One question on that is to me, books to music are pretty similar, you know, and I'm sure you can tell me why they're not. Was the real big problem when we jumped to things like electronics, you know, where it was more about comparing, you know, three CD players or the jump to maybe down the road. Things like clothing, where you had colours and sizes. And because even availability can impact search, right? If you have two items that are about the same waiting and the ones in stock and the other one's two week delay, all that must have just made the work so much more complex. So the beginning part of that question is how did the product mix change the search complexity because the releases seemed to accelerate.

You know, coming out of 98 and 99 and 2000. Yeah, I'll let Reuben talk about some details of that. But I think what I do remember is that we spend a lot of time trying to shoehorn in the attributes of music and video and toys and clothing into what we had at the time, into our existing Berkeley DB kind of index model. And I remember the details. But remember there's a lot of angst and a lot of, you know, unpopular decisions that were made about how we could possibly get this thing, get toys launch, get music launch. Just not invent a whole new database solution. Is that right, Ruben?

Yeah. And every product line tell us something new. I mean, even the transition of books to music was difficult because a book index would ship once every 1 to 2 weeks, maybe take a month. If there was some problems with the bill, music had to ship every Tuesday morning. And the reason why is because music was published on Tuesdays and so if data got stale, you know, we were unable to sell product and so this was every new product taught us something new. So we have an update our build scripts and learn how to deploy on a weekly fashion. And then there are some hits generated things where we had to learn how to publish on demand with some, you know, later product lines and dealing with all the different capabilities are how do you deal with different sizing and sizing requirements and search by that attribute when you're dealing with close. So a lot of it was just learning how to build a search engine that was capable of handling the infinite variety, including as much as like who's the vice president of the day here is a really important business decision. Somebody comes to the search engine and type in the word Xbox.

Do you show the item that's available or do you show them the item that's out of stock? Depending on who's vice president That day? The search engine had to be available enough to present the item that we wanted it to present some, some days. We wanted to present the item that was just about to become available and sometimes we wanted to present the item that is available now and not out of stock and depending on which category you're dealing with, whether that was an appropriate decision

enough. Yeah, that's a rabbit hole. You just feel like going down. Uh,

we only have a few more minutes. I had one more search

topic I wanted to ask about, particularly because Reuben, you have spent time at a nine as well, but I also want to hear Dwayne if you were involved in this, when did quote unquote sponsored, what do we call them sponsored links or pay on google? It's paid search, but we did start inserting that stuff into amazon results and now it's all over amazon results. Can you talk a little bit about that? And maybe before jumping right into this is a solution like what was the discussion around it? Like, is this something we should do and how to think

about it. Amazon launched its first word-based ad product in December 28 of 2000. So we had the notion of wife put ads on the site since pretty much the start of the website and I remember being part of that group because they asked me to implement it and it had led to one of the most important Jeff meetings I ever attended. I was in a room with a bunch of other vice presidents of categories, categories. And there were all adamant that we can't launch ads on amazon, we cannot launch out as on amazon, it's gonna cannibalize our sales, we won't do it. And they all unified under that concept as the engineer in the room. I just wanted to implement something. I thought it was gonna be interesting. Jeff Bezos walked into the room, sat down and said, we're going to be launching ads on amazon. I hear some of you have some concerns. The only one with temerity to speak was Jason Kyler and Jason said we have some concerns that launching ads will cannibalize sales on amazon.

And Jeff Bezos said, yeah, yeah, me too, let's measure it and find out. And that was it. The meeting lasted about 10 minutes tops. And it was one of those great moments where you get to see leadership and leadership that's willing to be data driven. And I was happy that happened. That product the slots product itself lived and died very quickly. It was not successful implementation because we didn't know what we were doing but eventually they figured it out and I'm sure the solution they have now is much more sophisticated than anything we could have drugged up at that time. Did that work

carry over two things you did at a nine? Probably a nine will be an entirely separate episode but maybe tease it a little bit like what where did a nine come from? Because in my mind again, not at a nine, it was all about search experimentation and innovation and blue sky thinking. Is that about what it was? Or was it also accommodating people that wanted to work down in Palo Alto. Like it was never entirely clear to me where that came from.

The answer is both, it was a way for amazon to learn how to set up a distributed office first and foremost really and search was led by Hoodie member at the time and moody joined the company. They asked me, you know, we be his mentor and given our relative seniority, I thought you know, I'm gonna be his river guide, I'll show you where the natives are friendly, showing where the water shallow and that ended up being awesome partnership and creating some really innovative technologies the help of Jonathan, Leblanc and Barnum endorphin and the rest of the team. And it was about creating a space for search both on amazon to be produced, but also to try innovations like Street View Block V, which was a precursor to Street view Do Search inside the book feature Search inside the book results, creating a way to Federated Cross Search results, which led to the creation of open search and bunch of other technologies that we're all about. How do you expand the accessibility of search and in an e commerce

world? This is really exciting because my teams launched. Look inside the book Melissa kurt Meyer was the product person. But again, my failing memory doesn't connect that with a nine doing all the real work. You know, the technology. Do you remember that side story where we took a couple 100,000 books or whatever it was? We ship them overseas? We had them scan and then we realized we screwed up and we didn't have them scan for OcR so we had to re scan them so that you guys could search indexing and search them if that rings a bell. But that's a true story

told the rings a bell. And it also let you know that this is also the foundation story of mechanical turk, you know, the OcR was bad and so for the exceptional elements, who would send them off to humans to take a look at it and somebody thought maybe you can make this a better system. And so mechanical turk ended being part of the launch of a nine because we were trying to get people center images for the block view. It's like in which image has Mcdonald's storefront in it. You know, you send it to a human to do that. As a result, you have to figure out which page which image was the best

looking image. I just keep writing down entire episodes here. So

I'm jotting down,

you know, search inside The book. It wasn't something that you should have been back about 10 times. Right. That's

Ruben

my feeling in the new co host. If I get there, you go, there you go

to the world. Well,

look, anything else you guys want to mention or you were thinking about before we spoke today that you wanted to discuss? Well, I did get some suggestions from both of you for some non profits and we'll link to them off of the post and show notes. Anything you want to mention about Woodlawn, Dwayne or your favorites ribbon. Uh, Woodlawn is just as a K 12 independent school in Davidson north Carolina where I live. And it's just, it's a little bit different than a typical traditional independent school. So check out the website if you're interested in it. And did you found that the school? Yeah. My wife and I founded the school in 2002. We've been working on ever since. We've done some teachings of coaching on the board. You know, everything you can think of. And it's been a great experience for me, awesome Ruben.

When I was working at Pinterest. Most recently I became associated with a group called dev color dot org. It basically is an organization dedicated to mentoring black software engineers and by other black software engineers. And I met the leadership of that team. We hosted them within the Pinterest offices down the barrier and up in Seattle and it was such a good organization. I really loved our partnership with them. And given how many, given the needs of the some of the software engineers we have in the Seattle office and down in the bay area offices. This organization is looking to make uh impact by having peers help their peers.

Well look, you guys thank you so much for being guests on the podcast and taking a flyer guests number three and four or 3, 3.5 personally. It was really interesting. Like said, I'm discovering this stuff, asking questions along the way. So it's, it's actually personally interesting and I think other people are going to like it too and it really is great to see you both. I've probably seen Ruben more recently than Duane, but it's also been a while. So you know, for the audience, thank you for listening to the podcast. If you'd like more details about what we discussed or you want to suggest an edit or future topics or guests, please visit invent like an owner dot com to sign up for the weekly newsletter. And this podcast episode in particular is going to have photos that Reuben shared with me old amazon t shirts,

wag puzzle pieces. And if you want to see what bottega results look like, we'll also have that in the results. And honestly, I still think they look pretty good. Um, you know, it's a pretty cool looking site. I wanted to invent like an owner site to look like the tabs, but my technical skills are very limited. So, and everybody be sure to subscribe to the podcast to get future episodes. That's it for today. And remember no sniveling. Okay. Mm.