Misleading benchmarks

Published originally on June 23rd, 2009

I came across a post on the Tap Tap Tap blog yesterday, in which John Casasanta marvels about the near four-fold increase in raw available power on the new iPhone 3G S, versus the 3G model.

It is rather startling to see the ease with which the blogger media and the iPhone community have accepted these claims as absolute truths. The iPhone 3GS is far more powerful than the 3G — without a doubt. But the benchmark we need to be looking at is the comparison between the iPod touch and the 3GS.

A brief history: The original iPhone shipped with a CPU capable of 620 MHz, underclocked to 412. It also shipped with a relatively low-powered GPU, the PowerVR MBX Lite 3D. After the original iPhone came the iPod touch, shipping with the same processing hardware. Apple then upped the ante last fall, with the release of the second-generation iPod touch. It contains a 620 MHz CPU, clocked down to 533, which is still a noticeable increase over the iPhone/3G. Also of note is that until the arrival of the 3GS, every device model contained a paltry 128MB of DRAM — the 3GS ups that to 256.

Now, you’re probably wondering why I skipped right over the iPhone 3G. It’s fairly simple:

With regard to hardware performance, there were no changes made between the original iPhone and the iPhone 3G.

Now there were definitely some hardware changes and software optimizations made for the 3G. I won’t deny that. iPhone OS 2.0 was released, with a laundry list of improvements and new features. Addition of A-GPS hardware and the 3G telephony chipset were certainly changes, and were (for the most part) improvements. But in terms of hardware performance, nothing was done.

So what am I talking about then? The second-generation iPod touch.

It contained the only performance upgrade since the original iPhone, and it was lauded as such when it was released. Since it was the current leading model in terms of performance, it is what we should be comparing the new flagship model to, and it is what we should be defining the new standard against.

Understandably, I am very much in doubt of these dramatic claims of performance increases since the "last generation" of devices.

A more accurate benchmark in "new" performance increase would be a comparison between the iPhone 3GS and the second-generation iPod touch. Fortunately, Daniel Pasco over at Black Pixel was kind enough to do just that, and his results support my arguments:

the iPhone 3G S ran about twice as fast as the 2g Touch in every test

Interestingly enough, according to his post, he is running his tests using the same code base which Casasanta tested on the iPhone 3G — development builds of Tap Tap Tap’s upcoming Plasma application, so we can say with certainty that comparing the two tests are "fair" (although that is not really what we’re aiming for either way).

In addition, Pasco ran some CPU benchmarks to see how much of the performance increase came from the CPU:

In these tests the 3G S comes out about 20% faster than the 2g Touch. Given that the clock speed of the 3G S is only about 12% faster, there is definitely some extra oomph coming from the upgraded processor architecture, but I think it’s pretty clear that the PowerVR SGX’s contribution to our performance increase is substantial.

In conclusion, while the iPhone 3GS is clearly once again redefining the standard for capability on a mobile device, it is only marginally so, and certainly not as dramatic as everyone would have you believe.