Attendees sat stunned at #SMX East as Baris Gultekin, Group Product Manager, Google AdWords, Google, Inc. clarified that the ubiquitous AdWords Keyword Tool now only provides keywords Google deems “commercial.” It’s been obvious anyway by degraded results over the last few weeks. This represents a shift of seismic proportions to those who utilize the tool for demographic research purposes, in and outside of AdWords. Here’s an overview of what we just lost and how to get the data anyway. [Author’s Note 1/1/2011: The KW tool has been updated. After reading this post, head over to our coverage of AdWords keyword tool fixes.]
Google owns the largest first-hand sampling of query data in many parts of the world. Now, breaking rank with 15+ years of holistic search engine open-source tradition, Google has decided marketers should no longer have easy access to the long, or even mid tail data (search inventory), unless the keyword has demonstrated “commercial” characteristics historically. Effectively, in tandem with the well-established user-herding “suggest” feature, this move consolidates Google’s semantic semi-monopoly into fewer keyword SKUs.
Sadly, this evolution has shut off our ability to mine much meaningful semantic data at all from the tool. This may be great for Google’s bottom line, or not. One might assume financial motivation. It’s bad for marketers who use the tool for a wide range of activities, none the least if buying AdWords and optimizing sites for Google’s organic crawl. Some would say it’s an incredibly selfish move by Google, yet more evidence that big G’ wants us to optimize AdWords accounts and organic pages, but not too much, too easily or too quickly without spending wads of cash first to figure things out.
This harsh reality has immediate repercussions for how marketers, from small business to large agencies, might leverage their understanding of Google’s inventory for search and business in general. Many posts will be written over the coming weeks illustrating just how lame the tool has become for serious research. The lack of detail in output is mind-numbing. Choose any query to test and it becomes readily apparent that there’s a lot less beef in the hamburger.
“Facebook” Must Not Be “Commercial” Then
Check out the keyword tool in action below. Google represents the output includes “all keyword ideas, including “adult ideas” and everywhere in the world. Could it be true? Do Google users really only articulate 12 semantic permutations of “Facebook” at phrase, broad and exact match? Eeesh… Obviously that’s a laughable proposition. These 12 keywords are what Google wants to sell as they productize Facebook related queries into AdWords inventory.
Trellians Keyword Discovery’s global premium database suggests nearly 2,000 keywords at phrase match, over 10 pages of output. Even keeping in mind that Trellian is working with a smaller sampling, it’s clear that our friends at Google have cut us off.
Not For SEO Any More?
For the better part of two decades, marketers have used paid inventory tools like Overture, Panama and Google KW tools to gain insight as to search engines’ inventory. Many of us thought these insightful oracles would never go away. After all, the engines need to show us their inventory in order to sell us clicks, right?
Retired Overture Keyword Tool
Paid inventory keyword tools have always been the easiest and most direct method to learn what users search for, those optimizing pages for organic search, used the data to define SEO. This is no longer possible with the AdWords tool, straight out of the box. Incredibly, Google has found a way to circumvent needing to reveal its inventory in specificity, even to those parties interesting in purchasing AdWords search PPC. This author never thought we would see the day where such inventory is not available.
Does this mean that big G’s KW tool is no longer reliable for SEO? We think yes. Though some might say that practitioners should now make a point of optimizing for people-herding suggest terms, there will be huge organic competition for these words. It is likely that this new short tail may be unattainable organically for all but the most authoritative and powerful sites.
This serves double duty for Google in that that many advertisers will be forced to buy more AdWords. This rush to purchase the limited “suggest” inventory may drive the cost up even further. For Google it’s a win, win, win and then win some more.
Not Good for Much of Anything
Savvy marketers have also used search engine paid inventory data for many on and offline purposes. For instance owners advise print ads on how people search. Companies and products are named based on keyword research. Universities design curriculum based on the subtleties of what students search for. Detroit uses keyword research to name alert lights in dashboards in the manufacturing design process.
The revolution of search has traditionally been centered on our ability to discover how humans search, the words they use, in what order, the frequency thereof and other subtle semantic permutations. Unless we want to subscribe (in resignation) to Google’s dumb-ass shortening of the known language universe, this tool borders on useless. No more will Google’s sampling of human queries be easily available just for the asking.
Dumbing Down Users & Content
Regardless of one’s philosophy of creating content, it might be hard to find an SEO who would suggest that only having access to the brain-dead commercial short tail is a sufficient enough a dataset to advise content creation. Remember the task is to write content both compelling to humans and diverse enough for optimization.
The suggest feature in Google’s search box, has herded searchers to Google’s prescribed short tail now for a couple of years. Though this data can be highly personalized based known and black box variables, the suggestions roughly correlate in their obvious banality, to the queerly useless results in the KW tool. It is a sad world when organization of the planet’s content is thinned for maximum profit.
There are other places to cull the long tail. This much is true. aimClear is switching to Yahoo and Bing’s combined inventory today. [Author’s note: Trellian, WordStream & WordTracker must be jumping up and down today going “yippee!”] Long live SEO. Here are a few analytic processes and tools, of many, to mine the mid and long tails.
Buy the Data From Google
It’s possible to purchase insight into Google’s mid and long tail inventory.
- Run Google PPC, at modified broad match, for the keyword tools’ output, in single keyword AdGroups
- Overbid, and only run the keywords just long enough to cull a large enough sampling.
- Take note of inbound paid queries using analytics. Many of these keywords won’t be revealed in the KW tool.
- Run a query report. The query report has never been fully transparent, in that not all queries are revealed. It is reasonable to think that Google will dumb this report down even further, since it’s a leak of inventory information.
- It’s important to note that there is no search frequency metric attached to organic analytics. This method gets us the keywords, but not the relative important of the KW, ability to sort by monthly searches, etc…
Microsoft has a real opportunity here. Since Yahoo rolled search into Bing, the sampling is much larger.
WordStream may have just become became more important. The methodology and toolset for mining keywords is an interesting approach that many.
Trellian Keyword Discovery is a classic alternative tool, with various databases. Many swear by the data.
WordTracker is a cool and campy keyword tool. Mined from meta search engine Dogpile and other sources, WordTracker yields quirky and powerful results from a relatively small sampling.
photo credit: wikioticsIan