” if you haven't read it my short review is go read it. Not now, but maybe in about an hour and a half. What follows is a text-based transcript of a session entitled "Inside Google's Search Office" which has Danny Sullivan, Amit Singal, Ben Gomes and Matt Cutts all on the panel. It was placed here from the KristineWirth.com blog simply because it is so large.
----- START TRANSCRIPT ----
DANNY: It's a real pleasure to be here and doing the panel for the Churchill Club. We've been talking about doing some kind of session about search for just over about a year and when they got close we began to talk about some ideas uh, this panel really came about from Steven Levy's excellent book “In The Plex: How Google Thinks, Works, and Shapes Our Lives
Um, and engineer David Bailey is quoted as being assigned to work in an office at Google and he's supposed to be working in this office with Amit Singal, Ben Gomes and Matt Cutts and he says it's definitely the “cool kids” office and that really struck me when I read that because I'd been to this office many times over the years I'd typically go into Google and have a days worth of briefings and then at the end of the day I'd usually end up talking to Matt on webmaster issues and we would be in this office and it would kind of turn into this search bull session.
Which is a lot of fun to hear all of them interacting about well, maybe the results should be this way or this particular issue. So this kind of session is kind of bringing that office to life in some of the things that go on with it. If I had to describe the areas that each of these men oversee when it comes to search Amit would be the brains, and Ben would be the looks and Matt would be the brawn.
So, I hope you're happy with your choices in life (laughter).
Amit oversees the ranking algorithm at Google, how Google decides what content should be shown in response to searches that happen; Ben oversees features that help you search better as well as the user interface look that lets you interact with Google search and Matt's the balancer, he's the chief of police he's the person in charge of keeping the people who would spam and pollute and harm Google's search results and bring disorder there under control. And uh, there each going to tell a little bit more about themselves and we'll start with Amit because he's the baby of the group having been at Google for only 10 ½ years.
AMIT: Thank you Danny so what Danny's referring to is the fact that of the three Googler's you are seeing up here I joined Google last. These two were already there and I've been there 10 ½ years. In a 12 year 12 ½ year at Google.
So that's the composition of the office we have been there for 10 ½ years together, 11 ½ and somewhere in there for Matt. And we've worked together since the day I arrived. I have had a fairly long academic background before I arrived at Google I got a Masters in search and then a PhD in search; yes, there is such a thing (laughter) and this is back when it was the sleepy field that librarians used to study and what computer scientists thought that really wasn't the prestigious thing to do; operating systems or compilers were and uh, this is all geek talk you know, and I ended up getting a PhD in search and um went to AT&T Bell Labs which became AT&T Labs to pursue an academic career and published a lot of papers and so on about search.
And then in 1999 my good friend Krishna Bharat who is father of Google news in a conference over beers told me that he's going to this company called Google in 1999 and Krishna was single then just to give you some color on the conversation and I and my lovely wife sitting there (motions) we had one child and a second one on the way and I said “Krishna, Google what?” like you are single you can afford to do this Google Schmoogle's all gonna die I have a family to feed I work for AT&T (pumps fists)(laughter). And next year I was here; working with Krishna and these wonderful people and the rest is indeed history. Ben?
BEN: So I think the common thing here is Krishna because I went to high school with Krishna and we both had chemistry labs at home and that was our common love actually doing chemistry experiments now this is something that both of us being brown-skinned boys don't do in this country any more (laughter) but that was our common thread that our common interest was and both of us went into computer science and he told me he interviewed at Google and there's a lot of sharp people there and it seemed like a fun place to work and I said “sure, why not, I can be there too.”
And so I interviewed there and I got the job and I remember my boss had soon asked me “Do you ever think Google will be 1/10th the size of Alta Vista? (laughter) I said “I'm not sure but it looks like a fun place to work and his boss made fun of me in the group meetings saying you're going to work for a company that has an exclamation mark in their name because Google if you remember used to have an exclamation mark and um, both of them now work at Google.
And the scuttlebutt at the time was that Google had a secret plan because it surely wasn't search; these are smart guys there's no business in search and so when I first joined Google I asked my boss at the time who is V.P. Of engineering “What really is Google's plan?” and he told me “It's search.” and I was so disappointed! (laughter) It turned out to work OK though.
MATT: Hey everybody my name's Matt and I was actually a computer graphics guy I was ah working on my PhD at the University of North Carolina Chapel Hill and got a job offer to go work for Google in 1999 my girlfriend at the time says “Well, that sounds like a really good opportunity but I'm not willing to move all the way across the country without a wedding ring.”
So we actually eloped and drove across the country, took a quick honeymoon, showed up and the very first project that I got at Google came about because my manager stopped at my cubicle and said “Matt, how do you feel about porn?” (laughter). And I said well, that depends why are you asking exactly?
And so they wanted me to write a family-safe version of Google which is called Safe Search so I wrote the first version of that and in the process of doing that I found out that there are bad people on the Internet. (laughter) and so for the past 10 or 11 years I've been dealing with bad people on the internet. And of course there's a huge number of good people as well but that's been a lot of what I work on.
DANNY: And Matt has a cornfield that he can wish those bad people into. (laughter)
Um, you don't want to be one of them. We're going to dive into some questions just a really bit of background just to kind of set the stage on some of the technical stuff it won't be that much of a technical issue but you know search engines the essential pieces of them there is a crawler that goes out and finds pages from all over the web and stores this content that the crawler finds in what we call an index. It's like a big book of the web. And the crawler's constantly updating, keeping all of the pages fresh some of them it visits a lot maybe every few hours some maybe every few minutes some it goes back as it needs to your personal blog maybe it will drop by once a month.
When all of this stuff is put into the index and we come along and we do our searches that's where the search algorithm kicks in. And it goes and it flips through that entire book if you will and it finds all of the pages that it thinks are most relevant and should be shown in response to a search.
Now for me and for others when you talk about what an ideal search engine should be and this is how the big search engines have generally operated they're akin to being a newspaper you have editorial content that is supposed to represent what somebody feels is sort of a fair what somebody feels is a representation of some of the best kind of stuff they can put together that is being shown to you independent of any advertising influences or any sponsorship that's going on out there. You can get ads but they might go off to the side just like you might get in a regular newspaper.
If you follow through with the newspaper metaphor search engines have a much more incredibly hard job than a newspaper. A newspaper will assemble this content each day their in their office building maybe their getting phone calls that come in their getting story tips or whatever for a search engine it's as if there are in the middle of Times Square. And they are surrounded by people shouting things at them saying “I got a great story for you! I got a great story for you!” And some of those are great ideas but the people don't shout very loudly and you need to kind of hear better and bring them forward.
Some of the shout very loud and their terrible ideas and some people are just purposefully being misleading and in the midst of all this you have people who are literally dumping garbage all around where you are working. Just junk all over the place. So go out there and come up with some good search listings off of that.
With that kind of background all of these things going on and the people actively trying to mislead you and people who just don't even understand that when they render their entire site built out of say Flash that it was kind of hard to read how do you figure out who to trust who not to trust how do you go about that core part of the job coming up with the right listings?
AMIT: That's a great summary of how search engines operate. Ah, Danny and imagine you have a book with billions and billions and billions and billions of pages. You do have an index like you have at the end of a book where which word appears where and that's the index of very similar index that most search engines including ourselves build.
Now when you type a query into Google you give us a few words and we use that index to find the few most relevant pages for your two, two and a half, three word query.
And this is clearly a deeply scientific process it's incredibly hard as a science as studied over four years and at it's heart there are simple heuristics simple heuristics like if you said "Danny" as your query then a page that contains the word Danny many times is probably more relevant than the page that contains Danny just once.
That's the first simple principle of search and then there are other principles like words like "the" are not really important so don't use them as boldly as you would use a word like "Danny". And you take such simple principles of how language behaves and you build algorithms based on those principles in an environment that's the modern web. And that’s the basic description of a search algorithm.
Now clearly in the real world everyone wants to come up number one for the query "Danny Sullivan" and therefore they give us pages which say Danny Sullivan 1,000 times whereas Danny is so polite he only uses his name once or twice on his website so in the real world all of these assumptions, all these principles are challenged like they are fighting through all of this complexity to get to Danny’s page (which we do get out there) so that's roughly how it works so we have taken simple principles of how language behaves and coded it into algorithms to return to you what we believe are the most relevant pages on the web that's the words of the web according to our algorithms.
MATT: And in the same way that all things in moderation according to the ancient proverb, the first time or two you see Danny Sullivan that's great but if you see it a whole bunch then it starts to turn a lot worse and so you have to realize that the web is like a giant torture test people will do crazy things on the web any program you write to parse a web page will break by the time that you get web pages that are long, that are complicated, people don't close their tables and also malicious or deceptive people and so you have to worry about not just ambiguity like there's a Danny Sullivan whose a race-car driver turns out but also just people who just want to rank for everything and so it is a really difficult problem but through those sorts of principles, those computer programs, those algorithms, rather than just having a single person saying I think this ought to be number one or I like this result for that, it's much more scalable to have the computer program s and say OK, they can work 24/7 they don't have to sleep it tends to work a lot better different languages all of those sorts of things.
BEN: I think one of the early sites that Google had was that the link structure of the web was going to be really useful for this and so the page-rank algorithm which was part of Google's early search was actually fundamentally leveraging the link structure of the web to actually try to find out which is the real Danny Sullivan and in addition to all of the other things Google's doing.
AMIT: So you can imagine when you're going on the web and someone says “Click here to go to Danny Sullivan's home page” that link hyperlink as we call it is a tiny recommendation for Danny and we take these people's voices which are encoded on these web pages as links and we build an algorithm to actually harness the power of the web these voices of authors who built this web to build what's Google today.
DANNY: Now, the links especially were the big kind of thing when Google came along people have been using links but you really took it up to a new level and suddenly you could find this needle in the million-page haystack that was happening.
And over this period if you go back to say 2000 it's largely what I’d say is the Google decade you were this unquestioned leader when it came to search and that was the gold standard and then toward the end of last year and through this year (2011) it's been really amazing amount of attention that’s been focused on search quality.
We have the sunglasses merchant who the NY Times profiled because he was convinced that because he was such a rotten merchant that all of the bad reviews were giving him links that helped him do better. We had complaints that people were saying “wow, I've written this content someone has copied it or scraped it and put it on their own website and now they're outranking me.
I'm the originator you're not finding my original content.” There were concerns that um, what's been dubbed content farms I won't get into the debate as to what those are but people were concerns that there were those content farms and I won't get into the debate of what these are but they were flooding Google with low-quality listings and then suddenly it seems liked there was this turn that Google was just terrible on all of their results are junk. And you've reacted to that with a series of updates and a series of changes in the merchant prompted this unprecedented five-day shift in your algorithm that I had never seen happen so quickly.
You've had other updates on the scrapers probably the most talked about thing had been this Panda update; Google often gives nicknames to when they make these algorithm changes and the most recent algorithm change they had was nicknamed “Panda” and um, so you're making these changes how do you decide what's broken how to fix it and assess whether or not your fixes are working?
AMIT: This is a very good question and the heart of Danny's question is how do you decide if I should change my algorithm in this direction and is that the right change or not.
And what we do at Google is that we have a few principles on which we operate this search team. And the first and the foremost principle that we have put in place is do what's best for the user.
And once you use that principle and develop science then to measure what's best for the user to the best of scientists ability you know I've been a scientist in this field for 20 years Ben and Matt have worked here for almost a dozen now so what we have done internally is develop very rigorous scientific methodology to evaluate if turning our algorithm right is a good thing or turning it left is a good thing and by how much is it going to do a good thing for the users or not?
And it would be really good percent of the context of the team. How we operate.
Let me just use a very, very simple example of how search progresses and I'm going to use a dumb example to just prove a point. Suppose one day an engineer walks up to me ans says “Amit, I'm going to promote all pages that have pink backgrounds.” Matt says “No! Blue backgrounds are better.” OK.
So now where does this engineer do with this idea the engineer goes and writes an algorithm that incorporates his or her idea which either promotes pink pages or promotes blue pages depending on who won the argument in our office OK, that's usually Matt. And then that algorithm is unleashed in a sandbox which we have of the entire web inside of our offices; not really in our offices; their in data centers but only available to our engineers no users are subjected to that sandbox and the engineer runs this blue page promotion idea in the sandbox.
So the new algorithm is blue page rank higher other page rank lower. Now you have the new algorithm and the current algorithm. The two ranking systems. We take these two ranking systems and put them through amazingly rigorous testing; rigorous testing involves things like we will take hundreds and thousands of queries from over past logs run the old algorithm, run the new algorithm.
We will show it to an independent human being outside who doesn't even know what algorithm is being tested and sometimes we do this on them (switches hands) so they don't develop a favorite left or right and computers keep track of either the left was the new or the right was the new and statistically after you have run thousands of queries through this left right blind test first number emerges that says, no, promoting blue pages is not such a great idea Matt.
MATT: Just a thought, you know.
AMIT: So there we go right? So that's the first test you have to pass once you have passed that level of testing we take a tiny subset of our live traffic and we take this algorithm and we put it outside in a real data center where users are being are doing their queries and we take a very, very, tiny subset of our live traffic coming into the data center and subject it to this new algorithm if it was good the blue one didn't make it to live testing.
And if users are now liking the new algorithm where their liking is described by they are clicking much higher in the ranks in that relevant documents are ranking higher and they are spending less time searching through results and clicking on multiple results they find the right result quickly with speed. Then the new algorithm's better.
And with these tests in mind and this much statistics behind every change and we run we make about 500 changes to our algorithm every year and we run 20,000 plus such tests every year.
With this level of rigorous scientific testing we put the statistical report together put by a statistician not related to the engineers his or her team that report is brought to a group of people all three of us included to make a decision whether this algorithm should be launched to the world or not.
And in that committee which you may have read about in Stephens books and articles is where a whole lot of debate happens it's not just these two numbers that will force you to launch something there are many more concentrations that go into launching the system like is it good for the web ecosystem at large? Would it benefit authors? Would it benefit high-quality content would it keep our system simple so that we can maintain it much longer?
And with all this concentration put together that committee says yes, we should launch this or not.
Hey, what did I miss?
MATT: You, you, very good description, ah, and there's a couple things you might not realize one is that there is no way that you can change a computer program to change how Google ranks and scores things that's going to be perfect, that's going to make every single query better.
Right because we get over a billion queries a day and if you get hundreds of millions of queries you could make five hundred of them great and only one worse but you're never gonna make all of them better. So there's always this tension, there's always this trade-off where you're trying to have the goals of the best results for users the best quality help in the web ecosystem and then there's also there has to be room for intuition and experience.
So for example, I work on spam that's people who try to cheat and it's definitely the case that users often click on spam because they see something that looks enticing oh this is what I looked for.
And so we seen examples where one person will click on the same spammer 8 times in a row and you just wanna “how can you do that?” after 7 times you couldn't tell this was not good stuff? Right? And so by the looking at just the raw clicks or just the raw statistics it might look like this is a horrible change so you have to take those factors into account but absolutely the statistics and those launch reports carry so much weight because they've built up this intuition about what are good changes and what are not as good for uses.
BEN: And in some ways this allows us to move a lot it is the intention for us moving faster because if you don't know that you're making progress you can constantly end up second guessing yourself and so on. So being very rigorous is essential to our speed of moving fast.
So we constantly sure that we bring the best thing we can for users at any given point. And we go to the same kind of rigor for uh interface changes too. Because those are sometimes even more complex to evaluate because there are more moving parts and systems. So the live experimentation and so on and there we have we augmented with user studies where we ah we have a usability lab where we watch people use the product and make sure that it actually works in practice. By watching users.
But they all go to the same level of rigor for all interface changes we make.
AMIT: So that's a long way of saying we are in the process of scientifically determining what's the best thing for the user. It this measurement in itself is a science yet it's a cutting edge of the science and that science is evolving as well but with the best of sciences ability to predict whether the change is good for the user or not we use that science to make changes to Google.
DANNY: I'll come back in a bit to the determining of the the relevancy that you've got it right. But Matt I want to pick up on what you mentioned on the spamming you are in this hostile environment um and what's the conference that you have it's the it's an information retrieval which is a science where people would be like how do we get our stuff out of the lexisnexis database that nobody's mentioning and then we have these hostile information retrieval?
MATT: yes there's an entire conference called Adversarial Information Retrieval (laugh) yeah, it's where the goals change because the people are trying to cheat and and deceive you.
DANNY: So you, you're in this environment how do you measure up what's wrong? Um, how do you prevent getting the wrong person you just said that you've got people who will maybe click 8 times on spam thing and you have to use some intuition and some people may click 8 times on the right thing so how do you do that?
MATT: Absolutely! So we try to provide very clear guidelines if you search on Google for “quality guidelines” we actually have instructions for publishers and webmasters about the sorts of things that are good and the sorts of things that are not as good and hopefully they make sense because we want to judge the same page that a user sees.
And so by that principle you shouldn't hide white text on a white background; you shouldn't show us a page about cartoons and then show hard core porn to users and the things that you would think would be intuitive right? Um, and so what's what's good and bad what's the curse is that once you know how to see spam you will always see spam. In any system you look at the cheaters you'll find people you'll recognize the people who are trying to game the system um, and the nice thing is that most of the time it's very clear.
Um, spammers tend to be lazy and they tend to go all the way out and try to get as much traffic as possible as quickly as possible and that leaves some footprints and some ways that you can spot and what's tricky is when people you know get more towards that gray zone so for example we have this category of stuff that we call web spam and we have very clear guidelines and some of the stuff that happened in the last year regarding content farms was stuff that you might consider just outside of the guidelines.
They didn't necessarily keyword stuff they didn't necessarily do something horribly bad for users but it was still really low-quality content that regular people would complain about. And so by by just stepping barely outside that zone it fell between the cracks for a little while and the nice thing about Google is that I've been in the same office you can turn around and you can say hey is this your job or is this my job?
Oh, okay, okay, we'll tackle this OK?
And that has worked very well it's amazing how at Google there's also the concept of a war room. We try not to go for big war-like metaphors but that one actually dates back a long time ago and whenever you have a crises you say OK, get everybody in the same room and then make such a difference for collaboration such a difference for teamwork because if somebody's a minute away you might walk three or four times a day to check in with them but if their in the same room or you can see through he glass walls and our offices have glass walls at Google you can see if they're at their desk you can walk right over to them you can see if they're looking unhappy looking at their computer and so you can say hey is something broken?
And that really does make a big difference in productivity as well.
DANNY: Now we um what you're saying is mercury news that had this article about hey I've been penalized by Google and I've had this big network and I didn't even know it happened and we've gone back and forth on this before I’m like why don't you just tell everybody you have a penalty you guys will tell people that some penalties but you won't tell everybody so why not just say hey, you know what you're doing something bad and I reported you in our Google webmaster central?
MATT: Absolutely so Amit talked very well about algorithmic search and the vast majority of what happens is all involving computer programs.
My team sometimes has to take manual action because if you type in your name and you get off-topic porn so you write a name or email to Google and say I would not like this porn result showing up for my name I’ve never been in a porn film in my life and we write back and we say well it's going to take us 6 to 9 months and we think that we might have an algorithm that might help that's pretty discouraging so my group is one of the very few where we actually are able to take manual action and so we've been trying experiment this past year where if we have taken manual action you can do what's known as a reconsideration request so it's basically an appeal and we will tell you whether we have taken manual action against your site or not now if an algorithm is ranking your site lower well I’m sorry with over 200 million domains there's no way we can talk to every single webmaster or publisher one on one we just you know literally everyone at Google would have to do customer support there would be no one left to actually run the computer programs and write new algorithms so we think that that's a relatively good compromise an that if there has been manual action you can now tart to get information about that you can say here's what's different here's what's new please let me back into Google I’ve taken off the hidden text.
DANNY: And if you're logging in has become more broad more...there's still some things we're not gonna tell you.
MATT: Well, previously we hadn't revealed everything we knew because we knew that there are some really bad guys out there like Al Gore has had his website hacked. Donald Trump has had his website hacked and so there are a lot of really malicious people out there who want to install malware, spyware...whatever you want to call them stuff you don't want on your computer and you don't want to clue those folks in so it is a tension but we've absolutely been moving more towards transparency as much communication as we can figure out how to do as a matter of fact some Google employees in this room have worked on how to improve that process and have really done a great job of it.
DANNY: Now people not doing well in Google has sparked complaints and in fact people not doing well in search engines have sparked complaints since before we even had search engines and there have been various things that have gone … the most common complain that I’ve heard in my time has been well you're not ranking me well because you're trying to get me to buy an ad.
But lately now it seems to be that the reason that you're not ranking people is because they're all competitors to you (Matt laughs) and to preserve the Google monopoly ...so what's the deal you guys are wiping out competitors?
MATT: So the nice thing about working in search quality is we don't worry about ads or revenue at all we have a very clear mission of doing what's best for the user so we've you know, that's not in our area scope that we're worried about at all.
AMIT: And so there's a clear separation between search and ads. No matter how much money and advertiser pays Google and that kind of goes into our revenue they cannot improve their ranking. Okay? That's fundamentally how it works.
And then the question that Danny poses is hey, haven't you now putting this stuff up because I'm competing with you and you have demoted me. Where this is stuff is say your stuff that's what they call it.
And I go back to our first principle do what's best for the user. Our job is to give users the answers to their queries. What they ask for is what we need to answer. Now in a most simplistic form if the user types the query 2+2 should we return a list of pages that have the words two plus two on it? Or should we say 4! (laughter) What would you expect? If you were writing the search engine what would you do?
Take the next example, when someone says the query 1658 Mountain View California would you not show them a map pinpointing exactly what they're looking for? Our job is to answer user queries. And that's what we do.
Everything that we do someone types GOOG they're looking for what's the stock price of our company today at that point. And we return that value right up there. When you work with our first principles that it's all about the user and our job is to answer your queries. Everything else just falls into place.
BEN: And I think we think about it in terms of the time it takes you to from the time to answer your query the time you get information you need and it's going to be a lot faster for you to see the number 4 over there when you type into in 2+2 so that is our goal to get the information the answer you need as quickly as possible and that guides we believe that's what's best for the user and that guides our decision making.
MATT: And if you go to the very extreme if someone comes to Google and types in “poison control” you really want them to get the phone number for poison control as quickly as possible so you want to get them that answer whatever it is they're looking for.
DANNY: So that gives that gets into some of the direct answers and then you get into this issue of people saying well, I just did a shopping search and instead of you showing me a you know, instead of you listing a bunch of shopping search engines your sending me into Google shopping or Google product search so now you're just trying to keep yourself there.
AMIT: This definition is actually somewhat absurd if you look at Google product search that you're talking about Danny it takes pages out there on the web it's just a search it indexes them.
It organizes that information better related to your task where your task is to figure out who much does something cost. How well is it rated? Should I buy this? Is this merchant high quality and Google product search takes information that's out there on the web and merchants can feed it to us for free. For free. There's no charge to anyone who is in Google product search they can feed us their prices their availability and so on and a the same time they can tell us what web pages their selling that item on so Google product search is just a different lens or a different interface on search that's far more effective for the query at hand.
And yes, indeed we send you first to that interface from there they can do their research and Google that merchant to complete their transaction.
BEN: In the end they are still going to Amazon or whatever merchant is out there.
AMIT: So I think calling you can squint at it hard enough and say this is Google's own stuff when the truth is it's all the pages out there on the web and merchants out there on the web feeding us information for free.
MATT: Well, and Danny you had actually made a really neat graphic a few years ago because at one point you might do a search for Tom Cruise and then if you want a picture of Tom Cruise you'd have to click on the tab, images and if someone types in sunset or daffodil or roses you might have learned over time that people actually want pictures of a sunset or daffodil or roses and so Danny had made up a cool graphic that was like Google in 2015 with like tabs all over the place look for people look for whatever and really what people want is they just want to type something in and get something useful back out. And they don't want to have you know like which of the 32 different options of the lenses do you want to search through so...
DANNY: Now, if you go to say the purist form of the search engine to me it's I’ve done the search, I’ve clicked and I’ve gone outbound and so from the search engine to a destination. And that the search engine itself is not a destination.
And when you talk about like the shopping search when I’ve heard these arguments...pulled up with me … ah yes, Google sent you from Google to Google shopping wherein you still have to go shopping and went to a destination merchant. But you get into these tricky issues where Google actually hosts content and indeed becomes a destination.
Um, cases like Google books or um, Google places where you're aggregating and consolidating a lot of information so if I do a search I rather than going outbound to the merchant I may go to places and perhaps the best example of this is YouTube where I do a search maybe I’m going to get a YouTube video that's coming up and you are a destination so there's inherent conflicts in that how do you deal with that how do you deal with those conflicts?
AMIT: So you know we deal with those conflicts with the same first principle that we've ever had test test test experiment scientifically test and make sure all your changes are good for the users. And once you have that principle in place and you're designing your result page because our job is to return a result page that's really really valuable for the query you design your result page in testing it extensively then for the query evolution of dance you would see yes evolution of dance site and yes you will see some YouTube videos as well because that's what the users are looking for. So we keep going back to our first principle whenever a conflict arises we go back to our first principle is this good for users? If it is, we do it.
MATT: Yeah and at the same time it makes perfect sense you know I remember Alta Vista used to be where you'd search for a person's name like Jeff Dean and you'd see “buy Jeff Dan on eBay” and you couldn't really buy people on eBay and so it wasn't a really good thing for the user experience if we were always showing something like ah every single time we showed a result from Google books that would be really annoying so we know it's not in Google’s best interest to annoy users to do things that are bad for users because then they get turned off there's plenty of other places to go and get information online.
DANNY: But as these other things have expanded have you ever thought I wish we didn't have that because then my life would be simpler because I wouldn't have to deal with these kinds of questions.
AMIT: (laughing) We like the questioning Danny. But fundamentally right, every time we build something OK, say Maps, it was the most innovative product when we launched Google maps the first product that allowed you to scroll the map ah in using Ajax technology at the time everyone else followed suit to the degree that now if you land at the page which has an embedded image of a map you try scrolling it right there.
So we feel proud to build these innovative products and then it we build these innovative products then giving users answers right on the results page is absolutely the right thing to do so clearly you know, our job is going back to our first principle giving users answers to what they ask for and sometimes we have to lean upon innovative products like maps.
BEN: He's actually ranking a variety of these innovative sources right? I mean it started with PDF when for the first time we started crawling PDF's in like 2000 PDF's are long documents and so they can begin to dominate all the results and so as soon as you started crawling PDFs you saw results full of PDF's so we have to deal with this issue … I was working on crawling and also ranking and then I realized well we just realized a whole expert on ranking maybe I should ask him and he had not even come to mountain view and I remember calling him up ts say Amit you did a thesis on this stuff what do you do?
And so I remember talking to him about this and... (points to Amit).
AMIT: So you know Ben calls me and he says hey we just started crawling PDF's and there long documents they have the word over and over again for every query I’m seeing just PDFs for the query IBM I'm seeing PDF, PDF, PDF for IBM.com and you have you have a written a dissertation on how to deal with varying document lens in search systems and I had joined Google in it's New York office back in 2000 and Ben was in California and this is around late 2000 he calls me up we were driving back home from Jersey City from having dinner at a friends house with our 2 kids in the back and my wife was driving the van and ah I’m talking to him over the phone solving you know giving him formulas that basically solve the document lens problem and that same time trying to keep the kids quiet take it easy guys it will be okay it's fine right to the degree that we didn't pay attention to what was happening to the car and we ran out of gas and had to call a tow truck. (laughing)
MATT: So maybe focus matters a lot you know trying to figure out these apples and oranges and how to blend.
DANNY: Matt you alluded to this earlier but you don't manually control the rankings um, if there's results that are showing up there is all due to the algorithm with the exception with a tiny little experiment that we won't get into right now that you just haven't done that so things show up and there have been times when you know the best example I think was one of the first examples where you would do a search for Jew and got this site called Jewwatch that came up that was sort of a hate site and people like how can you allow this and in the end you left it up and this goes on all the time there are these things here that a lot of people would have a consensus of saying this sucks. You shouldn't be listing that why don't you take it out so why don't you take it out?
AMIT: Let me take that right?
Um, Danny's referring to a principle that we have held dear to us in our search team which is that we would not manually promote, demote or remove results even if our judgment is saying our algorithms are doing the wrong thing.
The extreme case in point was query Jew when someone did that query it made all of us tremendously sad to see our algorithms fail. By putting an anti-Semitic site number one we were in pain we were really really hurt by this.
This is not what our principles are as individuals as a team as a company. And people said why don't you just black list that site? And get rid of them? However we said no, it's our algorithm going wrong and we will find a solution to this algorithm algorithmic problem by algorithms.
Because this judgment was so clear cut in this case that everyone’s instinct would be to black list that site but in real world not everything is such black and white. There are lots of slippery slopes. There have been many other queries where various interested parties have wanted us to shoot results out of Google’s search results and one such query is Scientology. Where we have both perspective return in our top 2 or three results and clearly one group doesn't want the other perspective to be there.
If we start intervening manually first of all we would make search arbitrary judgments; number 2, we … people over one … algorithm which would not fix similar problems in many more queries that our algorithm may be causing and in many more languages that we don't even read.
So our principle that we have held near and dear to our heart is we let our algorithms reflect the voice of the web and sometimes our algorithms could be imperfect and we work hard to make them better and better but our subjective judgment though mostly is right is not always the right thing.
And that principle has actually served us well over the last 10 years of us doing that and that is reflected in our users coming back to Google and liking what we do.
BEN: I think that's a principle that has come down from Larry and Sergey in fact for the query Jew shortly after the problem arose Sergey said we will not be changing this and then the web changed and actually the ranking changed and he came back and he said why did you change it and we said no we didn't change anything the web changed briefly back again but over time values have gradually improved to get rid of those many of those sorts of problems.
And side-effect of that that you referred to briefly is that we have been very good in languages we don't even speak because we rely so much on our algorithmic approaches because otherwise we have to do the same thing in every language an every country that we're in and that's an impossible task. A consequence of taking this algorithmic approach is that we can then be excellent in small languages in languages that otherwise are not considered maybe that important and it just works across the world.
AMIT: A normal company would set their priorities based on revenue in a language. We don't pay attention to that we say our job is to serve every query everywhere. In every language whether its in Swahili or in Hindi or in English. And this algorithmic approach allows us to dish the best results for English but the same algorithm since languages are not that different on in general when you dive deeper there are lots of differences that I’ll not go into right now but that algorithmic approach then any algorithmic improvement made in English not only serves English it serves Japanese, Swahili, Hindi you name it and entire search system in this world gets better and we feel proud to have had this principle.
DANNY: I'm sweating because I’ve got five questions in 5 minutes to get them in (laugh) so we'll go to the lightening round.
You know so you've got also listings that have impacts on people um Rick Santorum would probably be happy if you can make an algorithm change for searches on his name but you know you can argue well it's a public figure did certain things upset other people maybe you know that's just what happens but you've got people who are not public figures who are not known in any way do searches on say their names and they go I can't believe this things coming up and it needs to go away this is terrible. So why not let them take that stuff out?
MATT: It's a tough call because it's a slippery slope right; whenever you're faced with a he said/ she said situation you really don't have the time or the staffing or the judgment to be able to make the right call and every single situation so normally we say look Google is attempting to be a reflection of the web right if there's someone who is important in real life we want to reflect that on the web and you know if someone is … you in real life you can either send them a cease and desist you can get a court case there's lots of ways to take care of that and online there's lots of great ways to get your message out you can start a blog, you can tweet you can be on Facebook so there's many many ways that you can have search results that show up rather than just this one negative search result um but at the same time I would say that we feel the responsibility because we hear a complaint we hear people who are unhappy and it's safe to say with so many users and so many queries there's always someone whose unhappy or there's always someone who can point out a really bad search result and so I can't speak for these guys but a lot of the times I go to bed at night thinking about how could I fix that problem? Right?
Or you spend that time in the shower you know thinking OK so how is that going to get fixed but as long as you have hundreds of people behind the curtain who are thinking about how to improve search quality on all of those different dimensions than in general it tends to go pretty well.
AMIT: Let me just briefly add you know we have spent substantial amount of our careers devoted to search as students of search and we think day in and day out about search and how to make it better if someone is seeing some content about them that they don't like we hurt. But then there is that case when you're searching on a doctor who you learned had his license revoked three times surely the doctor doesn't like that result but our responsibility is bigger than that and that's the principle we operate under.
BEN: Because it always comes back to the user; what is in the best interest of the user.
DANNY: Now I talked earlier that you've had these attacks on relevancy and you know you've got to improve things and you've made these sorts of changes but one of the changes in it's not even new you've done it for a couple of years to try to increase the relevancy of results has been to personalize things so where someone is based out of if you're doing a search here versus down south you're gonna see maybe slightly different results and your search history the kinds of sites you been on can come in and influence it so we have these personalized results and now you've got people like um Eli Pariser and the The Filter Bubble
AMIT: So, you can just imagine a normal result for searching for a restaurant in Newport beach would be New York City restaurants OK that would be a normal results because by population and masses and assumption and usage that's probably what ... out. So we do personalize results and there are places like restaurants where we personalize them tremendously or whether we personalize whether results to where you are.
And there are other cases like the query “banks” guess what if I started returning large banks in the US you guys wouldn't be very happy or if I returned bank of America in the UK they'll say something snarky about us so there is that level of personalization which makes results tremendously relevant and then the type of personalization that Eli's filter bubble talks about is the view personalization that I’m just getting my view in first and the truth in search is that we are returning relevant results but it's our algorithms are so finely balanced to return some relevant results for you and some results that have the opposing point of view as you have seen time and time again with the controversies that we just talked about that our algorithms are tremendously balanced to give you a mix of what you want and what the world voice says you should at least know. I can give you a personal example the query “lords” you guys are thinking dungeons and dragons but I’m an Indian who loves cricket and there's this cricket ground in London, Lords, for me that's the first result however all the swords dungeons and dragoons are right there number two and three I would say that's an ideal page for me.
DANNY: I'll go with ya on that speaking of the cricket. Um, I’m going through Sophie’s choice which questions to throw away you, you all go now um I got two and then we'll go into the q&a and we should make it fit so we all know we not only have personalization of results we have socialization of results and which is sort of a subset of that personal who you know, you you're friends with and so on you've been using social search for a about two years now and having that influence things but now you have your own social network um so how is that going to have an impact in particular Google+ and the +1 system on the search results people are seeing?
AMIT: That's a very good question so Google was a pioneer in social search we were the first one to launch it based upon who you know we can bring their recommendations to the top of your search results sometimes or we can just bring them to your attention and this is great because if I’m searching for a plumber or a locksmith I would much rather have these businesses reviewed by one of you who I trust as opposed to the new york times square problem of everyone yelling I’m the best locksmith please hire me so social searches are very advanced in search technology and social search is based upon who knows who and who knows about what.
And then you put these two things together a very powerful system emerges. You basically marry the real world how the real world functions into search. And that search is more relevant because we can bring to you experts and sometimes experts in friends you may know who may know something about who you want to know that and that’s a much better system than it is today and Google + you know we would indeed Google+ has been a very successful system we're very excited about it and the Google+ we have much more data on who you know and what they know so that we can improve your search experience going forward.
DANNY: Have you notched up the social dial a bit since it's launch?
AMIT: Um, we experiment all the time with numerous things.
MATT: It's early days but it's pretty exciting.
DANNY: That's Google for you.
MATT: ...to see how it might work over time.
DANNY: OK, I'll squeeze in this last thing and then we'll go to the questions and answers and also we want to talk a bit about the team building stuff I want to catch you after the q&a come back to the things that have helped you I mean you started as a team of 10 people on a ping-pong table and now you've got hundreds of people all over the place and you still make it together so I want some ideas on what people can take from that.
But um, Ben you had mentioned a statistic when we were talking something like two humans only agree on relevancy 80% of the time and that in fact I think you had said this and that if it was um a year later and I showed you the same set of results that a year before you had rated you'd only agree with yourself 80% of the time so how do you how do you know if you're relevant? And then you mentioned humans and things that are out there but can anybody know and can we independently assess whether or not you're getting it right when you make these kinds of assumptions of this algo (?) will make everything better.
AMIT: Let me go to a sports analogy. In soccer you have tens of games in the season and if you lose two you are out. In baseball you have 160+ games in the season and winning or losing a single game isn't really going to determine whether you'll make it to the world series just look at the Giants.
And what we do is by giving thousands of queries and each query to multiple human beings we can statistically measure the inter-human being agreement of relevance that you mention which is around 80% in general you know I say this is relevant with 80% probability he will say this is relevant. So we measure we can measure that and incorporate that inter-human agreement into deep scientific measurement that I’ve talked about and once you know that then you can bring up statistical tests to say have I run enough samples through to definitively say with 95% confidence things like … and so on that the new system is indeed better.
DANNY: Does that skew a bit because you ask these people who are not experts on the things that they’re' searching for to tell you whether or not they think things are relevant.
AMIT: But time and time again this 80% statistic speaks for itself so for your query I'm 80% accurate and for your query you're 80% accurate and just knowing that and incorporating it into our evolution formula allows us to do the right thing.
BEN: And the live data we have huge volumes of live data where we do tiny experiments and so there you can really make use of strong statistical techniques to make sure you are getting the answer right and then the person is whose looking has a real information need and so you get to uh much closer to a good determination of whether or not they're satisfied.
MATT: But I think the part that might frustrate you is that Google has spent 10 years trying to figure out how to evaluate potential changes but we haven't talked as much about that outside of Google and so you know to have a third party to say the relevancy is better of this search engine or that search engine is really difficult and unfortunately a lot of people will do like three queries and that will say OK I found my home page therefore this is the best search engine and so there hasn't been as much rigor in the third party analysis a lot of it is reduced down to metrics about who happens to have more growth in queries this month or something like that.
DANNY: Um, ok we're going to to to the open Q&A and I'm also going to ask you about where you think things will be in 5 years from now and we'll get to that in the last 5 minutes there. If you have a problem with ranking on Google you can move to this side of the room (laughter) and we'll get those questions to you so we have our first question over here.
PERSON #1: In the last year or two I’ve seen more and more of what I'll call type-ahead where Google tends to finish my typed query and I think I’ve seen search ahead where I actually get results you know before I’ve hit return can you comment on you know, how that's worked from your perspective and where it's going?
BEN: We started doing Google auto-complete about 4 or 5 years ago and the idea there is that we still think of the search process in terms of time of from you having an information need too the time you seeing the information you want. And so auto-complete actually helps you formulate the query because you get the query formulation wrong if you've got a spelling mistake if you've got the wrong words it's very hard for you to get the right information right? So auto-complete is the first step in helping you formulate your query and essentially greasing the rails to your answer.
Now the second step is the best way you can evaluate whether this is the right query is if you can see the results. So the idea behind Google instant was to begin to show you the results as you type. So you can begin to evaluate whether your partial query is what you wanted and we find that like people get relevant results well before their finished several seconds before they've finished typing their query. And so we're seeing like a lot of benefit to users from actually showing them results early and we we very very pleased with some of the impact it has on the time it takes to users to get the information they want.
AMIT: Ben is being modest he led Google instant and Google instant has been a huge success for us we are very proud of what we have done here.
MATT: And in fact Google kind of obsesses about speed to the point where we've just launched something where if we really think it's likely that you'll click on the number one results then we might pre-render it so by the time you click on that result it shows up instantly you don't even have to wait for that to download so there's hundreds of people at Google thinking about every aspect of the search session how can we like give you bionic arms to search faster and bionic brains and all this sort of stuff and the goal is to get you to what you want as quickly as possible.
AMIT: Yeah they didn't buy my Google helmet idea.
DANNY: Next question?
PERSON #2: (difficult to hear) ..once you've done your query … ah (inaudible)...how does it affect you knowing the competition authorities including the FCC are in fact looking over your shoulders and knowing how expensive and time-consuming it can be for the company, damaging your reputation, ...that it's always a roll of the dice … what is it you hesitate to do what is it you don't that doesn't get done because of this concern?
MATT: That's a great question I think having worked at the company for 10, 11+ years the core essence the principles of the company have remained the same because it's almost like spiderman with great power comes great responsibility and I like to think that we've operated with the idea that someone could be watching over our shoulder and tried to make the right decisions all along.
Now you're right it can be a roll of the dice and so some of the things that we've been doing is trying to educate people how to search work you know what are the different policies so for example we've made over 450 different videos that talk about common questions and answers and if we can do a good job of explaining yeah there is some manual stuff whenever there's spam but the vast majority of it is algorithmic and explain how these are best practices and how every search engine does them to some degree so hopefully that decreases the odds that there will be some mis-communication and some fluke will results in a problem we have a very good legal team and they think a lot about all of these issues but I like to think that the the core values at Google are such that we try to behave as if we were the underdog we try to figure out how can you step into the other person's shoes and how can you have a good appeal process how can you scale up on communicating how can you scale up on transparency and communication but you have to be mindful that you know as an engineering company we always have to have the idea that we can be wrong we are very disruptive company in a very disruptive space which is technology and you have to be mindful of that responsibility and if there are people who have suggestions for ways to improve you absolutely want to be open to that listen to that whether that's someone you meet at a conference, someone who is tweeting to you or someone who is a regulator.
DANNY: Next Question?
PERSON #3: Hi um, I had a question about what you do to like the quote unquote bad guys like for instance um I understand searches are like generators as a result of the algorithm but like for instance JCPenney a couple of months ago a NYTimes article about how it can game the system like what did you do to to affect their rankings and like what can a company like JCPenney to gain in the rankings after wards.
MATT: Yeah, absolutely so some of the Googlers who worked on that exact thing are a couple of tables away from you as a matter of fact right now so for people who didn't read the article this was JCPenney had engaged an SEO company a search engine optimization company and there are certain things that Google prefers not to see in our search results so if you are buying links that pass PageRank it's almost like a form of payola it's like you're paying someone to say nice things about you but you're just paying them to link to you and so that's something that’s against our guidelines just like you're not allowed to do payola on the radio and so whenever we see ah, a a violation of our guidelines like that we are willing to take action we absolutely try to write algorithms to spot that counteract those links diffuse and detect them so that they aren't an issue sin in the first place and strangely enough we already detected those links several time I think that there was a failure in that particular case in that we didn't escalate and take stronger action earlier but then the flip side is once the company tries to clean things up and JCPenney did a very good job of trying to make sure that they were doing ethical SEO after that incident happened then you have to make allowances and say OK how do we let you back into the search results at an all times the goal is to try and make sure you're ranking appropriately in the search results so it is a tricky problem in that in some cases judgment does come into play whenever you are looking at violations of our guidelines but the vast vast majority of the time you want those to be handled with the computer programs with the algorithms so that you don't have to bring that judgment into play.
BEN: Google in general is to reflect the real authority for that website the goal of search in general its not to be ahead or behind it and when somebody is trying to game the system to be ahead of to be ahead of the real authority in the world that's when we take have to take action.
DANNY: Next question.
PERSON #4: Thank you all for a great panel so far ah, Ben Parr of Mashable. So you had this product this cool product called Google real-time search we you could be searching getting real live time updates and feeds through twitter , facebook etc., then the twitter deal expired and they disappeared and this is a question from my Google+ ah, audience by the way, so the question is when or if is Google real-time search coming back and do you need twitter for that or is that going to come primarily from Google + data?
AMIT: So, ah what Ben is referring to is we launched real time search based upon data from various real-time data generation ah information generation systems like Twitter, Facebook, Fan Pages, and so on and our deal with Twitter expired we didn't come to an agreement and after that ah we decided that the value that product was providing was not … for users and we took it offline. And we're actively working on as we speak figuring out using our current G+data and other sources that are out there to revive the same functionality into Google search results so I could say stay tuned we are working hard on it.
DANNY: Why didn't we get and why don't we have now just the ability to search Google +? I mean your a search company and we don't have the ability to search your own social network.
AMIT: (laughs) you're feedback is very well taken.
Believe me. And ah …. and again we are on it.
MATT: It's fair to assume there's always lots of things we want to do and a finite amount of people so...
DANNY: Next question
PERSON #5: With the addition of +1 search results does this offer a new way to game the system where people can +1, +1, and get their search results to the top or um how does that work?
DANNY: Yeah, you can buy those now or so I've heard. (laughing)
MATT: Not to be cynical but every change in search involves a potential way to gain the system I’ve gotten a little jaded over the years but the thing that's actually nice is if you think about ranking as it existed a few years ago it was primarily based on links, anchor text, what's on the page so you know we have over 200 different signals the idea that the web might move from an anonymous pages and some dark corner where you don't really know who wrote what to a web where you actually do have some sort of annotation that says this person wrote this or this person vouched for this page and you can have the reputation of individual people or authors so if someone who is a NY Times columnist shows up on a forum and leaves a two line reply, that could be really important even if nobody links to it so it's absolutely the case that people will try to game social signals, Google+ all of these things you already see people trying to sell you know +1's but there’s different ways where you have new signals and new ways to intersect that and hopefully prevent that and the idea that you can get a big win from all of these potential new signals is absolutely worth giving it a try so.
AMIT: And no signal is used in its absolute form every signal is absolute form has it's shortcomings like you mentioned for +1, we cross hundreds of signals to build what's Google today.
MATT: You can certainly imagine having fun saying oh we'll buy some +1's and then see what shows up (laughs) you can play games like that.
DANNY: The +1 data right now it is used as a ranking signal for when your logged in correct?
AMIT: Ah, when you're logged in it is a social signal if someone you know has vouched for something indeed it's part of our social signal.
DANNY: ...and if you're logged out is it being used as one of many signals?
AMIT: (nods) we are experimenting with numerous things always.
DANNY: Which is Google’s … for yes.
PERSON #6: So um my question comes from the part of mage and video searches so you know if I was to search for a moonlit beach and if the images or the video wasn't tagged with those specific words I was doing some advanced the art of video and image search which can get beyond the beaches without those words.
AMIT: So our image search algorithm is far more sophisticated than just looking for those words either in the caption or nearby on the web page. There's a lot of computer technology built into our image search algorithms some of the team members are here today ah, which basically use those images to say hey this one looks like a beach and so does this one. And now you have a positive loop you can find some great images of beaches and then you can find some other images of beaches that didn't really say beach or didn't say beach with the same density as our algorithm would have liked so it's a good question we use a lot of that technology it works very well and we're constantly improving it as you saw with the our most recent launch of search by image an image comes to user ah what's that you just drop it into Google image search and guess what the very high likelihood that we'll find it for you what you're looking at. Using the same version algorithms.
DANNY: Next question?
PERSON #7: I come from the mobile industry and I'm sure that ah mobile phones have different kinds of search queries ah be interested in a comment on that but next thing that we think is coming is machine to machine mobile communication and I’m wondering if you're preparing for machines to be generating queries in that way or ah, and do you think how different do you think machine queries will be than people generating queries?
BEN: Well I think the first point is not actually that accurate the mobile query stream is getting to be more and more like the web query stream a while back it was different when phones were really slow and interfaces were not for web browsers the query stream was very different but today is gradually approaching exactly the same distribution as the desktop stream and so to answer that part of the question um I think actually the things are much more similar that you think.
AMIT: And let me also add to that part of the question with the innovations we have made at Google with things like word search where it's hard to type on a mobile phone so we give you voice search – amazingly accurate voice search and you start seeing human beings behaving the same as they behave on desktop I will say that a mobile device we still get somewhat more local oriented queries and for that we have launched numerous innovations like on Google’s home page itself there's a list of hot things you can do there's like restaurants, coffee shops and so on and so forth so mobile query stream is actually reflecting what users need which means its coming closer to the natural distribution of queries and mobile has been a great success for Google with all the innovations that this search team has made in mobile not only by voice search but thinks like our buttons are more touchable on a mobile interface our maps are far more designed for this tiny interface we have done immense amount of work on mobile and that reflects in our success in mobile.
Now the question about machine to machine searching right now we haven't observed that much of it happening ah, and I think once that system picks up we would have to analyze how that looks but in an ideal world of search for me, search would be so accurate that you can just type a query and the machine should assume that the search engine the other machine would answer it correctly and then then build a whole ecosystem on top of that platform and that's our dream platform that we're building together.
DANNY: Next question.
PERSON #8: This may be a follow up on the bionic arm question or comment there which is I’ve heard people in Google talk about search-less search which I interpret to mean that maybe you can even take the personalization to the extent and anticipating what the user might want without even typing a query so I’m curious about you know your points of view on that and what are the challenges that poses to you.
AMIT: So it's it's a great ah question because this kind of technology is what we as kids dream of right you know computer would tell you what to do and the truth is...
MATT: Maybe not tell you what to do but suggest help you understand what might be possible.
AMIT: And uh and the truth is first of all you can see that in an in a future that's possible based on pieces that we already have you can build systems where computer can help you tremendously in making your far more efficient human beings so my computer my phone already has my calendar it knows when I’m free, it has my to do list which says I have to buy a baseball mit and it has a map based on Google’s local of all the places that sell and it knows where I am so it's not too far out there that you can imagine that computer can gently prompt me hey, please do pick up baseball mit you are three minutes away from sports authority right there and you have 30 minutes free on your calendar.
MATT: And by the way it's kind of annoying I have to have (applause) it would be nice that I have to have a to do list at all right because Google has announced something called Google wallet and wouldn't it be great if you could go to the grocery store and you could buy things with your Google wallet all you do is tap to pay and then over time if you wanted to and gave permission Google could say well you haven't bought cat food for 6 weeks normally you have bought it every 4 weeks do you just want to add cat food to the list and then finally I don't have to think about oh I need A1 or I’m out of salad dressing or whatever like all these little traces we leave if you're willing to opt in for those kinds of things Google could be the little tap on your shoulder that's like hey don't forget to get wet cat food or something like that.
AMIT: And the key there is that users have to opt-in to these things. It's a critical aspect I’m a human being who deeply cares about my privacy and that's the key part everything we do at Google we think about that sometimes we get it wrong we stand up and apologize and move on but that's how these systems and technology will evolve if someone told me 20 years back that you would type into some machine what's the height of mount Everest and it would spit out an answer I’d say you're smoking crack buddy go on.. but see what's possible with technology we all have to dream it together and then build it in the privacy...
DANNY: If only you guys could answer our email that would be a real solution. The next question?
person #8: Brian Fox Western Union, ah first of all Google has to know that my cat died last week because I blogged it and you shouldn't send me that… (laughter) … that'll be your job um I’m curious about your vision your um image recognition technology do you use the power of the people doing the queries when they confirm that the beach really was what they were looking for? Does that loop back into your algorithm and improve it's intelligence.
AMIT: Numerous factors go into deciding in image search what images are most relevant and most liked by users and there is that positive loop that based on users choices of what we return we can improve the system going forward so when you combine the power of our search algorithm based on words with the power of vision algorithm and this wonderful loop that's when you get what's Google great image search today.
PERSON #9: … also with Western Union we're a 160 year old company and I’m intrigued with the dichotomy of culture which I care about so your obsessed with statistics and algorightms and yet you talk about glass walls and the power of being one minute away from each other and being physically co-located can you talk about that in this global world and web and blah, blah, blah all that stuff thank you.
MATT: Yeah absolutely it's been enormously helpful to be very close to the relevant people and Google has offices around the world and so for example numerous people in our office will have a video conference unit right there where the glass wall they can look behind them or they can look through the glass screen and talk to someone in New York and so with very little work it's easy to bring up somebody and collaborate from Tel Aviv, from New York, from Zurich and that makes a huge difference you know you always have time zone s but just being able to have that face to face connection makes a huge difference you know short-circuit an email conversion hop into some sort of face to face communication it really saves a lot of time and a lot of misunderstandings.
AMIT: And we have found this proximity of teams it makes us tremendously efficient we can turn out things as Danny pointed out in one case in 5 days we could launch an algorithmic change at the scale. And this proximity doesn’t always have to be physical and by video conferencing in our office there are like two or three video conferencing units that are open to the world all the time and people can just ay hey can you un-mute I need to talk to you and there we go we have a conversation.
BEN: And I think in the history of Google we went through periods where we were very densely packed and we also found and that as not by design is just that we hadn't gotten more space but we also found those extremely efficient in the company where people were backed into an office and they communicated a lot more than they otherwise would have. It might not have been their first choice based on their background many people came from backgrounds where you had your own office and so on but clearly the kind of energy that I don't think arises otherwise without that kind of density.
MATT: Yeah, there’s some companies where every developer has their own office or as Ben ah there was one office that had three Ben’s in it and so they called I the Ben Pen right? So but it really does make a difference to be able to just turn around and we have all these queues you can do heavy duty video conferencing you can do a hangout in Google plus now, you can do Google chat and there's these queues that are subtle well I’m red but you can interrupt me if its really really important all the way down to email and meetings and all sorts of things so having that spectrum...someone whether it's something that's a quick hit 20 seconds or a half hour meeting that really makes a big difference...
AMIT: in the early days we used to say we pack them tight and give them deodorant. That's how it works.
DANNY: And we've got time for one more question.
PERSON #10: Shilas from Citrix so ah thanks to the panel for wonderful insights search so my question is about when I do a search most of the time I get results back saying that 30,000 results in 2 seconds or so but thanks to excellent algorithms and principles first two or three links my results many times so why bother spending time for finding and searching for the 19,000 + results and giving to us kind of to save time there.
AMIT: So you have seen that numbers sometime appear on Google's result page that says we have 20,000 or 200,000 results but the truth indeed is what ...said is that if you haven't found what you're looking for in the top 2 or three results which shouldn't happen that often or you can send us mail um then really going down further is not that useful it’s just that our algorithms do compute numbers for all those results and so we give a user the user an indication of how much there is but that doesn't mean you have to read 20,000 and feel pained about that.
BEN: I think in some ways it's a historic artifact and so sometimes you'd get a misspelling you would get much fewer results than the real query so we would use it as an indicator it's no longer true today we correct your spelling and so on but it's a historic artifact. So I think a little bit of … too.
MATT: And by the way I’ll give you one tip is that it is an estimate it's not an exact count so if you ever notice we only give three significant digits when we guess how many results there are that's a little Que to let you know it’s not really 982000 it's roughly 982000...
DANNY: And I’m curious I know I can still do a search ...actually get a bigger number.
MATT: Right so rough estimate (laughter).
DANNY: The last two things I want to come back to the team aspect so you've talked a little bit about it already where that you can network whatever but what …
MATT: So there's one metaphor which is the baseball metaphor which is when you've worked together for so many years it's almost like having that many games in a season you don't get that frustrated if you lose one time or if somebody tells you to go back to the drawing board because I remember when ever I launched safe search this porn filter the first time I was already to go I was all ready to flip the switch and two engineers tested it out and said you have too much stuff labeled as porn that really isn't porn ad I had to go back to the drawing board and figure out how to make it better and at the time I was kind of frustrated about that but over the course of doing many many many many many launches you build up that trust to where you say this person I looking out for the best interest of the user this person Is looking out for the best ...of Google so take that as cont...don't get so caught up in one particular battle one particular controversy because it's guaranteed tomorrow there will be a new point of discussion and that’s helped a lot in making things more collegial.
AMIT: Yeah and now I look back at my 10 years of being part of this group we started around a ping pong table with 10 people and now we have hundreds of people in our group and unwittingly somewhere we developed a principle that has made this team out innovate every other team out there and that principle was I would put leaders in place who I respect technically so the entire management hierarchy of the crew is built of people like ourselves who were engineers wrote code and understand what an engineers going through their happiness their pain.
Then entire group hierarchy built from people who have worked in the group many many years hundreds of people report to us now but every one who reports to us has managers who have been there many years. The whole group leadership is built off purely technical people with deep technical knowledge about search and deep understanding of what goes into this innovation machine that is Google.
And that we just put in place early on because I couldn't find enough people to manage people as the group was growing so Matt you manage 10 people okay so Matt got 10 people and then Ben you manage 20 people and that's how we grew the group and it has served us very well this is a lesson in leadership that we have all learned in retrospect we wouldn't designing for this but it so happened that if you are in the innovation space you need to make sure that your leaders are so technical that everyone that works for those leaders respects them as technical people. And that's worked very well.
BEN: And I think all of our leaders all stay with technical titles and they think of themselves first and foremost engineers as we do too even though they do a lot of management they first they self perceptions as technical engineers.
DANNY: Just … by the way your title is Senior … are you Senior or...
MATT: Principal yeah?
BEN: Google fellow.
AMIT: My title's also Google fellow.
DANNY: Which pretty much has nothing to do with what you actually do.
MATT: Add Google you can get anything you want printed on your business card. Literally they do not care you can there was there have been some pretty crazy business cards printed right because you know titles are kind of at least at Google they don't make as much sense to obsess about right and so you can get whatever you want on your business card then you don't fixate you don't obsess about it you worry more about the job at hand and that tends to work well.
DANNY: And then in a twee-table or Google plusable or facebookable short statement where are we at in 5 years from now other than bionic arms doing all of our searches.
AMIT: So you know all of us have our views of where we want to head five years form now I was raised on a healthy does of star trek and I want my star trek computer OK that's what I wanted five years from now I should be able to talk to it ask it whatever and it should be able to have a conversation with me and searches of course fundamental to that.
MATT: I want star trek in five years but in one years I want the ability to get reminded that I need to get cat food while I’m on the way home and also the ability Google voice recognition has gotten very good and it can't be that much harder to make a well punctuated email so I want to be able to do my email while I’m driving home I’m talking at a stop light or whatever and have it look as if I very carefully crafted it that can't be that hard of a problem that shouldn't even be a three year problem so I’ll go back and file a bug when I get back.
BEN: Yeah I think I share Amit's vision to Star Trek compute but I think in the shorter time frame I want to search on my phone which is with me all the time to work really really smoothly and effortlessly and it's not quite there yet it's gotten a whole lot better right voice recognition...you can … these amazing things but like you can take a photograph I was in south America recently I could take restaurant Spanish menus where I took a photograph of the menu and with image recognition and translation I could then translate the menu in English I was like wow that's amazing and it's sort of kind of you can sort of see where you're gonna go but you're not there you're not there yet and I want that reality to become completely fluid completely reliable so that people all over the world can actually you know communicate and get information really easily I think this matters not just here but pretty clearly in other parts of the world where people don't have as easy access to information where people in India and people in Africa and people in the third world really have...information the way that we take for granted in many ways today. And they have access and I think...lives and I think that will change the world in many ways.
AMIT: So let me communicator...search by saying I feel like like going to candy store every morning and you haven't seen nothing yet.
DANNY: well it sounds like all search in the future will be mobile and we won’t even be searching it will be happening for us so um thank you all very much for being here it's a really been a delight I wish I had another three hours to keep going so...(applause).