Step 1 : Get Lots of Links By The Enraged Faithful
Blaze, an Ottawa-based web optimization company, drew from the play book of easy publicity by publishing the results of a benchmark pitting Android and iOS against each other in a webpage loading deathmatch.
Android was the victor, hitting the finish line (the onload event) an average 34% faster (2.1 seconds versus 3.2 for iOS).
The test measured the elapsed time until the onload event fired loading a selection of Fortune 1000 websites, which as a generalization average towards being simpler sites that are less exploitative of recent advances in web technology.
More important still, the actual comparison was between the Nexus S and the iPhone 4 as proxies for their respective operating systems. The Nexus S being the 1Ghz Hummingbird-processor equipped Samsung device pitted against the Apple iPhone 4 running a very similar processor — the A4 and the Hummingbird are twins separated at birth, both designed by Intrinsity and Samsung — at a slower 800Mhz.
All else being equal, I would expect that in this matchup the iPhone 4 would lag behind by a good 25%.
Of course all else isn’t equal: the operating systems are completely different, and the browsers greatly diverged from their shared WebKit roots. Yet a number of comparisons of recent incarnations of Android and iOS on similar devices have found their browsing performance converging: In most cases iOS gives the sense of progress quicker (showing partial pages earlier), while Android tends to complete rendering first. Android has made big gains from 2.1 to 2.2, and again from 2.2. to 2.3.
The Android 2.3 browser is fantastic, and is an absurdly responsive platform. Indeed, the whole operating system has gone from being a endlessly halting stop-the-world garbage collection demonstration, to a smooth as butter, refined UI.
So in this case an Android device with the latest OS on a faster processor comes out on top. No big surprise, so no big controversy,right?
Hardly. Turns out it was a pretty big controversy.
The iPhone Loses This Shall Not Stand!
It turns out that these results riled up a hornet’s nest.
And on, and on, and on. Pretty extraordinary how reactionary the defensive hoardes are whenever Apple or the iPhone comes out the loser (look at some of the enraged comments below the Blaze pages — this is an absolute embarrassment to the technology world, and these raging defenders are the reason online comments are seldom of value or insight). There have been countless ridiculously flawed surveys covering the Android and iPhone ecosystem, and they are trusted and repeated verbatim so long as the iPhone is the beneficiary.
I think my favourite ridiculous defense came from Michael Gartenberg (the go-to guy if you’re looking to fluff up some empty tech story)-
“Consumers don’t buy smartphones by going into the store and timing web-browsing experiences with their stopwatch,” Gartenberg said. “I don’t think this has any impact on the buying public, per se. The headline becomes a lot more misleading than the actual performance because performance gains of less than a second just don’t matter.”
What a silly statement: The desktop browser wars have been waged on the battles of milliseconds here and there. Android’s biggest deficiency prior to Gingerbread (2.3) was interface halts that lasted generally in the order of 60ms or so, and this was something that virtually every user rightly complained about. If these results held, and one device imposed a one second+ tax on each and every page load, that is very significant.
Benchmarks Are Always Flawed
The benchmark is far from perfect. Immediately I would question the choice of target websites: The top websites that people actually visit, particularly on smartphones, have little correlation with the Fortune 1000, and the former will tend to be far more dynamic and leading-edge than the latter. Alas, most benchmark are deeply flawed.
Says Daring Fireball-
It’s easy to see that Mobile Safari is faster than UIWebView — just run something like the SunSpider benchmark twice, once in Mobile Safari and once in any app from the App Store with a web content view. On my iPhone 4, Mobile Safari runs SunSpider almost three times as fast as an app using UIWebView.
I think there’s a general misunderstanding of the benefits of JIT compilation, especially in the context of a test such as this.
Go ahead, completely turn off JIT compilation (about:config,filter on jit and toggle all of the .content items to false) and see what the impact really is. You’ll find that it’s still a shockingly speedy browser, even though it is suddenly a Sunspider weakling.
Addons gain from JIT, though of course that is completely immaterial to an embedded browser control on a mobile device.
Blaze Wins Regardless Of The Criticism
I commend Blaze, however. They played the media and community like a fiddle, and their PageRank will benefit heavily as a result. They do offer an interesting service which many of us were completely unaware of before.