Posted on June 11th, 2007

wpIn early June, Google updated their webmaster guidelines and included specific intent to kill WebPosition as the poster child for Google’s disdain for automated queries. Since WebPosition has been best friend to many (including me) for years, it was a disappointing development.

However before we really get going here, it should be noted that organic prominence reporting, WebPosition’s forte, is dead anyway. The inevitable progression of personalized search, which returns customized SERPs based on individuals’ web history, codifies the deal. Bulletproof conversion and ROI metrics calculated by modern analytics are the only uncontested measures of SEM success.

Clients Understand the Web 1.0 Way.
Many of our search marketing clients still perceive organic keyword rankings as the singular measure. To an extent they are correct in their assessment of value: top rank for a keyword is still a great asset. However, true search marketing experts teach prospective clients the new reality during the sales process and lead existing clients through the adaptive process. Also, WebPosition has other features which remain useful like Link Defender and Page Critic AKA “Beat the Algorithm.”

WebPosition’s Organic Prominence Reporter: The Pathology of Demise
The announcement from Google was cryptic: “As of December 5, 2006, we are no longer issuing new API keys for the SOAP Search API. Developers with existing SOAP Search API keys will not be affected.” SOAP was the Google API feature which allowed access to keyword rankings directly from the Google database infrastructure. On December 6th the SOAP API ceased to function in WebPostion, even for those of us in possession of API keys.

WebPosition blamed it on Google and advised users to turn off SOAP access in WP and switch back to the illegal screen-scraping mode. You see, in the past Google has officially discouraged screen-scraping the SERPs.

Since WebPosition Gold™ and other organic rank reporting and sponsored search bid management tools have always relied on immediate access to current organic and paid search rankings; this was a major blow to life as we knew it. Using WebPosition Gold’s “Reporter” was now officially against the rules.

Organic Prominence Reports
The SOAP API key was the only method Google encouraged for us SEM types to automate testing a keyword’s organic prominence without scraping the SERPs. Though the rules were not rigorously enforced, the abandoned capability left no legal method to test Google keyword rankings. SEM practitioners were left out in the cold and stuck with the sole option of breaking Google’s rules or running blind.

Most of us screen-scraped anyway and Google seemed to gradually clamp down. Even with settings that emulate human search patterns, automated searches now result in Google shutting you off with an oblique error message accusing your computer of being infected with spyware or some nasty virus.

Automated Bidding Software
Search Engine Optimization was not the only search marketing agency tool to take a major hit that day in December. Automated bid management tools like Apex Pacific BidMax suffered too. Formally, current sponsored ranking positions could be instantly accessed allowing bid management software to make bidding “decisions” based on actual current rank. With the demise of SOAP backend access, bid management tools entered a new era of forced dependence on average sponsored keyword positioning as opposed to actual current position…bummer.

With the API changes, Google’s AdWords Position Preference, and Preferred Cost Bidding features, SEM pundits speculated that third party software bid management through the API had evolved to a place of dwindling value and might spell the end for bid managers using software from the paid search bid management companies like Atlas, Omniture, KeyWordMax. Time is proving this true.

Google Kills WebPosition
In early June, Google updated their webmaster guidelines with specific intent to finish off WebPosition as the poster child for Google’s disdain for automated queries:

“Google’s Terms of Service do not allow the sending of automated queries of any sort to our system without express permission in advance from Google. Sending automated queries absorbs resources and includes using any software (such as WebPosition Gold™) to send automated queries to Google to determine how a website or webpage ranks in Google search results for various queries. “

and

Don’t use unauthorized computer programs to submit pages, check rankings, etc. Such programs consume computing resources and violate our Terms of Service. Google does not recommend the use of products such as WebPosition Gold™ that send automatic or programmatic queries to Google.

Yeah, So What?
At the end of the day this is no big deal. It’s true that in this transitional phase, where personalized search has not fully hit, WebPosition is still a useful (albeit illegal) SEO tool. Clients still relate to organic prominence reporting so search marketing companies like us still profile sites with WebPosition, especially during the sales process. However WebPosition and other tools which depend on real-time reporting of organic and paid keyword rankings on SERPs are all the way dead. Google has done their level best to limit transparency and our ability to use tools (other than Google’s own) which depend on real-time feedback about current keyword rankings.

Google Accepts Adwords Cash From Banned Product.
As an aside, it should be mentioned that that Google has no trouble whatsoever accepting AdWords money from WebPosition to advertise an SEO tool which Google has specifically banned.

wp-adwords

and

wp-adwords2

Subscribe Today

We'll keep you updated on the latest Aimclear musings & appearances

  • Scott

    Hi Marty,
    That message has been on Google’s web site for years. This is not new at all. Numerous SEOs and SEMs, web master forums, etc. have posted on this topic for the past several years.

    2004:
    “…For ages, I’ve not seen Google list pages from WebPosition. It’s probably banned, but of course Google doesn’t confirm these things. And as a searcher, it’s not something that was disclosed to you.

    The Google-WebPosition problem? Google doesn’t like the burden the popular rank checking software places on its system, so explicitly warns people not to use it. But ironically, you’ll notice that Google has no problems taking ads for the software from WebPosition’s many resellers…”
    http://searchenginewatch.com/showPage.html?page=3344581

    WebPosition wrote about this back in 2004 as well:
    http://www.marketposition.com/blog/archives/2004/01/dispelling_the.html

    Also, WebPosition is not illegal. There is no law against download search results from an engine. If anything, it is the same thing that Google does. Google sends an automated tool to your site, downloads your content, then re-publishes it on their site via their engine.

    WP allows you to to go to Google’s site, scan their content, and presents it to you for your personal use. What you do with it, is up to you.

    Google’s bread and butter is advertising. If some web site owners can track their rankings in order to formulate a search engine plan, gain some advice on improving their pages, and obtain some successful rankings, they may not need to advertise so heavily on Google. Remember that Google is a for profit, advertising supported company. It is not just a “free” search engine, out to index the world’s content.

    It’s sad that they have turned off their Soap API. It is not -just- WP that has been affected. There were a lot of really great mash ups that used the API in order to create brilliant web sites.

    http://radar.oreilly.com/archives/2006/12/google_depreciates_SOAP_API.html
    http://battellemedia.com/archives/003212.php

  • Marty Weintraub

    Thanks for the history. What’s stark now (and why there’s more buzz about it in blogs at the moment) is how WP was singled out by Google as an example in the new guidelines. Google got pretty serious about naming names and WP is poster child…bummer.

    THANK you for your post. I hope Google figures out that programs like WP SELL more advertising than the cost of the extra system load.

    Marty

  • Scott

    “I hope Google figures out that programs like WP SELL more advertising than the cost of the extra system load.”

    I’m all in agreement.

    Even with some decent rankings, there are always going to be keywords that are entrenched and nearly impossible to rank well on no matter how much you optimize, gain links, etc. You will NEED Google’s advertising programs for those keywords. Even for results where you have a good ranking, some times it is strategically helpful to have multiple listings on the same serp where you already rank organically. Multiple chances to be clicked on -and- you have better control over the text that displays in the paid adverts.

  • Clickfire

    I still think Google needs to offer a way for search firms to get organic prominence reporting without violating the terms. I wonder if they might offer something like this in Google Webmaster Tools with all the enhancements we’re seeing.

  • Marty Weintraub

    Whatever tools come into existence, they have to take into consideration personalized search. The most reliable way to measure organic prominence is by the TRAFFIC a keyword drives. That’s the only sure way to know.

    Thanks for stopping by clickfire. 🙂

  • Asad Hanif

    But Many peoples still using webposition gold.

  • James

    Hi Msrty,

    I’m just looking into ways of extracting data from a list in the natural serps. Is there now a way to do it that is acceptable by Google?

    If not, are there any companies that offer an alternative database to Google or would a search engine like duckduckgo offer an SOAP API?

    Thanks,

    James

    • Marty Weintraub

      Not sure about duckduckgo. There is no extraction that Google sanctions. Tools like mozenda work, but violate TOS.