I love to stay up until early in the morning playing Werewolf. In early December I went to a journalism conference called “News Foo Camp” in Phoenix and played a lot of Werewolf. When I got back, for some reason I searched for [werewolf] — maybe I was thinking about making a custom deck of werewolf cards. Because I was dogfood-testing Search plus Your World, this is what I saw:

In the top row of pictures, you’ll see a bunch of people playing werewolf, including a picture of me as the werewolf in the top-left image. Doing a generic search like [werewolf] or [photos] and getting back a picture of you or your friends is a pure, magic moment.

Let me tell you how it happened. I have Brian “Fitz” Fitzpatrick in a circle on Google+, because he’s in charge of Google’s Data Liberation Front and he’s an all-round awesome guy to boot. Fitz published an album of 25 Werewolf photos shortly after the conference. Okay, but I’m only in one of the 25 pictures; how did Google return the picture of me first? It turns out that Brian had tagged me in that single photo.

Once you know the trick, it might not seem like magic anymore. In fact, this is the “things just work” experience that everyone in the tech industry strives for. But when I searched for [werewolf] and got back a recent picture of me playing werewolf, it did seem like magic right then. I suspect as more people take Search plus Your World out for a test drive, they’ll quickly experience similar magical “Aha!” moments like I did.

I was reading some of the comments on tech blogs, and I wanted to clarify something: Search plus Your World does surface public content from the open web, not just content from Google+. For example, look back up to the top-right image from my screenshot above. That’s actually a werewolf photo that Gina Trapani took and it’s hosted on Flickr, not Google.

Here’s another example. If you follow the excellent and erudite Jennifer 8 Lee and search for [general tso’s chicken], Google can surface this high-quality thread from Quora:

By the way, that’s a fantastic thread for Google to highlight, since Lee literally wrote the book about General Tso’s Chicken. It’s exactly the sort of “just works” user experience you’d want.

It’s not hard to find content shared on other sites. For a search [grand unified theory of snack food], Paul Buchheit shared a link on FriendFeed, and Google can highlight that:

Or if I search for [connectbot], here’s a link that Brad Fitzpatrick shared on Live Journal:

(Yes, we do have both a Brian Fitzpatrick and a Brad Fitzpatrick at Google. People sometimes mix them up, but they’re different.)

I hope that helps to make my point. Search plus Your World builds on the social search that we launched in 2009, and can surface public content from sites across from the web, such as Quora, FriendFeed, LiveJournal, Twitter, and WordPress.

The team should be finishing the rollout of Search plus Your World in the next day or so, and I hope you enjoy it. Remember, to see the new results, you’ll need to be signed in with a Google account and search on google.com. Give this new feature a whirl: once you see how much better personal search can be, I don’t think you’ll want to give it up.

The first fake-Matt comment I remember was over Marketing Pilgrim around November 14th, 2011. When Frank Reed checked out the fake comment, it came from 74.120.13.132, which is an exit router for Tor. That means someone went to some trouble to hide their tracks.

The second not-Matt comment was on November 18th, 2011. The impersonator wrote:

Normally we do not comment on ranking methods but I’ll explain a misconception: input from manual raters is used only in the rarest of cases when a non-brand cracks the top ten for high value money terms.

The tone (and content) of the comment was so far off that Matt McGee questioned whether it was really me, and I was quickly able to clarify that I never wrote that comment.

The third one I’ve seen was just a few days ago on Search Engine Journal, and included gems like

[Google is] very transparent. Some sites do not even have an address listed, yet we have everything, including the credit card numbers for adword advertisers. That is a strong signal for us to list them ahead in organic search as well.

The claim that “Google ranks AdWords advertisers higher in our search results” is fake and untrue; it was one of the first myths I debunked when I got online.

The web isn’t built to prevent impersonation. On many places around the web, anyone can leave a comment with someone else’s name. So if you see a comment that claims to be from me, but makes crazy claims (e.g. that we preference AdWords advertisers in our search results), let me know. I’m happy to verify whether I wrote a comment, e.g. with a tweet. Thanks.

Okay, I sometimes get a question about whether Google will always use the url from rel=canonical as the preferred url. The answer is that we take rel=canonical urls as a strong hint, but in some cases we won’t use them:
- For example, if we think you’re shooting yourself in the foot by accident (pointing a rel=canonical toward a non-existent/404 page), we’d reserve the right not to use the destination url you specify with rel=canonical.
- Another example where we might not go with your rel=canonical preference: if we think your website has been hacked and the hacker added a malicious rel=canonical. I recently tweeted about that case. On the “bright” side, if a hacker can control your website enough to insert a rel=canonical tag, they usually do far more malicious things like insert malware, hidden or malicious links/text, etc.

I wanted to talk today about another case in which we won’t use rel=canonical. First off, here’s a thought exercise: should Google trust rel=canonical if we see it in the body of the HTML? The answer is no, because some websites let people edit content or HTML on pages of the site. If Google trusted rel=canonical in the HTML body, we’d see far more attacks where people would drop a rel=canonical on part of a web page to try to hijack it.

Okay, so now we come to another corner case where we probably won’t trust a rel=canonical: if we see weird stuff in your HEAD section. For example, if you start to insert regular text or other tags that we normally only see in the BODY of HTML into the HEAD of a document, we may assume that someone just forgot to close the HEAD section. We don’t allow rel=canonical in the BODY (because as I mentioned, people would spam that), so we might not trust rel=canonical in those cases, especially if it comes after the regular text or tags that we normally only see in the BODY of a page.

But in general, as long as your HEAD looks fairly normal, things should be fine. If you really want to be safe, you can make sure that the rel=canonical is the first or one of the first things in the HEAD section. Again, things should be fine either way, but if you want an easy rule of thumb: put the rel=canonical toward the top of the HEAD.

You can find out more info and apply to be a search engineer at Google if you’re interested.

What I would not recommend is sending tons (as in, thousands or even tens of thousands) of individual url removal requests to the url removal tool. And I would definitely not recommend making lots (as in, dozens or even more) of Webmaster Central accounts just to remove your own urls. If we see that happening to a point that we consider excessive or abusive, we reserve the right to look at those requests and responding by e.g. broadening, cancelling, or narrowing the requests.

So if you’re sending huge numbers of requests to our url removal tool, it might be a good idea to take a step back and ask whether you should be removing at the directory level instead.

Recently I read a fascinating essay that I wanted to comment on. I found it via Ars Technica and it discusses “search neutrality” (PDF link, but I promise it’s worth it). It’s written by James Grimmelmann, an associate professor at New York Law School. The New York Times called Grimmelmann “one of the most vocal critics” of the proposed Google Books agreement, so I was curious to read what he had to say about search neutrality.

What I discovered was a clear, cogent essay that calmly dissects the idea of “search neutrality” that was proposed in a New York Times editorial. If you’re at all interested in search policies, how search engines should work, or what “search neutrality” means when people ask search engines for information, advice, and answers–I highly recommend it. Grimmelmann considers eight potential meanings for search neutrality throughout the article. As Grimmelmann says midway through the essay, “Search engines compete to give users relevant results; they exist at all only because they do. Telling a search engine to be more relevant is like telling a boxer to punch harder.” (emphasis mine)

On the notion of building a completely transparent search engine, Grimmelmann says

A fully public algorithm is one that the search engine’s competitors can copy wholesale. Worse, it is one that websites can use to create highly optimized search-engine spam. Writing in 2000, long before the full extent of search-engine spam was as clear as it is today, Introna and Nissenbaum thought that the “impact of these unethical practices would be severely dampened if both seekers and those wishing to be found were aware of the particular biases inherent in any given
search engine.” That underestimates the scale of the problem. Imagine instead your inbox without a spam filter. You would doubtless be “aware of the particular biases” of the people trying to sell you fancy watches and penis pills–but that will do you little good if your inbox contains a thousand pieces of spam for every email you want to read. That is what will happen to search results if search algorithms are fully public; the spammers will win.

And Grimmelmann independently hits on the reason that Google is willing to take manual action on webspam:

Search-engine-optimization is an endless game of loopholing. …. Prohibiting local manipulation altogether would keep the search engine from closing loopholes quickly and punishing the loopholers–giving them a substantial leg up in the SEO wars. Search results pages would fill up with spam, and users would be the real losers.

I don’t believe all search engine optimization (SEO) is spam. Plenty of SEOs do a great job making their clients’ websites more accessible, relevant, useful, and fast. Of course, there are some bad apples in the SEO industry too.

Grimmelmann concludes

The web is a place where site owners compete fiercely, sometimes viciously, for viewers and users turn to intermediaries to defend them from the sometimes-abusive tactics of information providers. Taking the search engine out of the equation leaves users vulnerable to precisely the sorts of manipulation search neutrality aims to protect them from.

Really though, you owe it to yourself to read the entire essay. The title is “Some Skepticism About Search Neutrality.”

Just a reminder: please leave your question on the Google Moderator page, not in the comments here. When you leave a question on the moderator page, people can vote for the questions and I can see which questions people are most interested in.

I’ll repeat what I said last week. Do you remember when you first started on Twitter, and you didn’t know quite what to do with it? Who do I follow? What do I say? I didn’t really “get” Twitter for months. But as I found interesting people to follow and got the hang of it, I began to see the appeal of Twitter and started using it more often. I’ve noticed Buzz is tracing that same trajectory for me: an initial burst, followed by a bit of a slump, and then a steady climb as I found people that make Buzz interesting.

Buzz fits nicely between tweeting and blogging. Twitter is perfect when you want to share a link or a single crystalized idea. But Twitter isn’t as strong for group discussion or expressing medium- to long-form ideas. At the same time, blogging is great when you want a permalinked url that will stand the test of time, but it can be a real pain to write a blog post. I always feel like I have to polish my blog posts and it seems to take me at least an hour to write a blog post no matter what I say.

Buzz has the casual feel of Twitter, but you can dive into a topic pretty deeply. Buzz is easier than a blog post, but can look almost as polished. I find Buzz especially good for asking opinions, because the signal-to-noise ratio is (at least right now) quite high. I think Buzz is incredibly strong for internal company discussions too, so I’m looking forward to Buzz rolling into Google Apps.

If you haven’t checked out Buzz, or haven’t checked it out recently, you might want to give Buzz another look. You can follow me on Buzz if you’re interested; we’re having a nice discussion about favorite Chrome extensions right now.

I believe encrypted search is an important option for Google searchers. The Electronic Frontier Foundation (EFF) has asked for secure search in the past (see this post from 2009), and I credit them for helping to put this on Google’s radar. Another inspiration that helped to spark this project was Cory Doctorow’s book “Little Brother.” It was one of my favorite books of 2008 and while I won’t go into the book’s plot here, it’s a quick, fun read. “Little Brother” also makes a compelling case for encrypting HTTP traffic on the web.

Some people don’t yet fully understand how SSL search works. I saw one commenter say “If they still pass in the search parameters in the URL (Get), what’s the point? People can still see what you queried, if they made them “post” messages it might actually do something.” It’s important to realize that even though you as a surfer can see the query in the url, the sites between your browser and Google can’t. Google OS demonstrated that by sniffing a regular HTTP query and an HTTPS query in Wireshark to show that the query can’t be seen going over the wire.

Thanks to all the people at Google who did the all the hard work and heavy lifting to deliver this. One of the main engineers behind the effort was Evan Roseman, a member of the webspam team who you might have met at previous search conferences. In fact, Evan was originally scheduled to be on our site review session at Google I/O this past Thursday, but we decided that launching SSL search took priority. I also wanted to say thanks and congratulations to the other Googlers (for example Andrew Widdowson and Murali Viswanathan, but also many, many others) who generously gave their time and effort to make the launch happen and happen smoothly. You might think that switching on SSL for websearch is easy, but for a website with the complexity and scale of Google, it’s really not. The launch wouldn’t have happened without a ton of assistance from Googlers from many parts of the company, and I sincerely appreciate it.

I hope you enjoy https://www.google.com and find it useful.

By the way, if you’re attending Google I/O you’ll probably want to install the very spiff Android app for it. You can search for [Google I/O] in the Android Market. And if you want to know what to expect in the SEO site review session, here’s the video from the panel we did last year:

If you see me at Google I/O, please say hello and tell me what you wish Google would do that we’re not doing.