Cloak…..and dagger your website
by Robbie, February 6, 2006
The story of Google yanking the German BMW site from its index is running rabid around the net. In a nutshell, Google doesn’t condone the practice of displaying pages to users that are different than the pages it indexes (this is referred to as “cloaking”). In the BMW example, the page indexed by Google was full of keywords to help increase its page rank while the page seen by users was much more scaled down and did not contain as many keywords.
Sidestepping the issue of Google acting as the sole judge and jury, the cloaking issue is a big deal for internet publishers. Google drives a huge amount of traffic, but it can only do that for content it indexes. I haven’t talked about business models yet, but assuming a subscription-based service, premium content would have to be password protected which prevents Google’s bot from indexing it. So you could have a lot of great content, but if it is password protected, Google will never see it.
The alternative is to customize the response based on the requestor (i.e., cloaking). You could give search engine bots unlimited access to the site while requiring authenticated access from users. But if Google finds out, they’ll pull the content out of the index and you’re screwed.
Currently search engines are missing out on a significant amount of valuable content (perhaps some of the most valuable content since publishers are trying to make money from it). Last summer there was speculation about Google testing a premium content search feature, but to date nothing has been formally announced.
Making premium content available to the search engines is one of the issues that keeps me up at nights.
Some argue that it’s far from clear whether Google would find the following practice “cloaking”:
Let’s say you have installed the vBulletin forum on your website and you have restricted viewing access to members who have registered and logged in. However you wish to permit the Google spiders to crawl your forums to include posts in their index and also permit the display of relevant GoogleAds. So you create a hack which detects whether the visitor is a Google spider and if it is, full viewing access is granted.
Now the question is this: if the spider sees different content to a (unlogged in) visitor is that cloaking? The logged in visitor will see the same content as the spider. Google have been typically evasive in response to questions about this issue but it is a major issue affecting many site managers as they have the conflicting goals of maximising exposure and GoogleAds revenue, and retaining privacy/member rights over parts of their sites.
Denby April 15th, 2006
What you described is exactly the issue I’m referring to. Your example would be considered “cloaking” based on my understanding. i.e., presenting different information to spiders than what an anonymous user sees. Google doesn’t make any distinction between good cloaking and bad cloaking. Check out the Google Webmasters Guidelines.
Robbie April 16th, 2006