Lazy Surveys Enable A Lazy Press

On Monday I posted an entry titled Lies, Damn Lies, and surveys, the focus being that surveys — especially online surveys — are often of dubious merit, and worse are often intentionally or ignorantly misinterpreted by the press.

Yet it works wonders for both the press and the marketing research companies: There is a whole industry of lazy tech writers who will run with whatever is sent to them. Any slant they provide merely globs extra vaseline on the lens of accuracy.

In that case I was spurred by some just released results of an online survey done by GfK.

Take a look at the Fortune summary of said survey. As is the case with most other retellings, the Fortune writer seemed to simply rewrite the original Reuters story which itself said:

The survey found that just 25 percent of smartphone owners planned to stay loyal to the operating system running their phone, with loyalty highest among Apple users at 59 percent, and lowest for Microsoft’s phone software, at 21 percent. Of users of Research in Motion’s BlackBerrys, 35 percent said they would stay loyal.

The figure was 28 percent for users of phones running Google’s Android software, and 24 percent for users of Nokia Symbian phones.

The Fortune story includes a big graph with the title “Plan to stick with your smartphone OS?“. The iPhone towers at 59%, with Android down at a miserable 28%.

Wow, looks pretty rough for Android! For anyone whose work relies upon smartphone trends, this is pretty big news.

So I emailed GfK and they kindly responded with the same press release that they had sent out to the press. Unfortunately it still is only a summation, including no notes on methodology or the actual questions asked (which can often be very leading), yet it is far better than most survey companies that simply hide their summary behind a paywall, enjoying the attention as the press mangles it into something more hit magnetic.

Here’s the table that was the source for most of these stories.I have to guess (based upon the surrounding wording) that it was the responses to a question asking current smartphone owners what they’re going to consider when they next upgrade their phone.

Smartphone Ecosystem
Overall Apple Nokia
(Symbian)
BlackBerry Windows
Mobile
Android
Will stay loyal to smartphone OS 25% 59% 24% 35% 21% 28%
Will stay loyal to smartphone OS
but switch handset make
7% 8% 16%
I will look at all options 56% 36% 60% 58% 61% 49%
Will switch smartphone OS 6% 1% 8% * 5% 2%
Don’t know 7% 4% 7% 6% 4% 4%

There’s Android at 28% among these single choice options. That’s the meat of a lot of easy “news” stories.

Note the second line, though: “Will stay loyal to smartphone OS but switch handset make“. Sum those two lines and for Android suddenly you’re at 44%. Sure, maybe someone with an HTC Evo 4G is looking longingly at a Samsung Galaxy S, or maybe they’re imagining getting that Android-powered Playstation phone when it comes out, but we (meaning virtually every single press reference to this survey) are talking about the OS here.

So Reuters (and any that followed Reuter’s lead) couldn’t manage to achieve a grade-school reading ability. I’m serious: my daughter had table reading assignments in grade 1 that were just like this, a job that many in the tech press would fail.

Extraordinary.

But it gets even better. Among those who answered “I will look at all options” (which ideally should be 100% of respondents), here are what each current platform’s users are considering as their considerations when they upgrade at some point in the future (multiple choices obviously being allowed).

Current Smartphone Owners
Overall Apple Blackberry Nokia
(Symbian)
Windows
Mobile
Android
Apple iOS 53% 85% 46% 47% 43% 38%
Blackberry OS 33% 22% 74% 34% 41% 22%
Symbian 23% 14% 18% 40% 18% 13%
Windows Phone 7 41% 28% 40% 47% 65% 31%
Android 51% 40% 41% 48% 42% 84%

It paints a rather different narrative than the common,egregiously wrong interpretation. You end up with hilarious ignorance like this nonsense. With what you have learned here, check out the chart here and see if you can figure out what’s wrong.

The iPhone still takes the lead (with loyalty oddly dramatically higher in Germany — the home country of the survey organization —than in any other nation), but look back at the original Reuters statement: “The survey found that just 25 percent of smartphone owners planned to stay loyal to the operating system running their phone…the figure was 28 percent for users of phones running Google’s Android software“.

I suppose that sounds better than the more accurate “44% of Android users have pledged undying allegience to the operating system, regardless of the endless and unpredictable changes among competing choices. Of the remainder, the vast majority still expect to hop back on the Android train.