Wednesday, October 02, 2013

Everyone cheats at benchmark scores, except Apple and Motorola: Report

Samsung shouldn't be taken to task, at least not exclusively, after having its benchmark optimizations on the Galaxy Note 3 exposed. The key word there is "exclusively," as with the exception of a couple of notable OEMs, everyone cheats, AnandTech reported on Wednesday.

Of course, that doesn't make it right. In addition, the degree of "optimization" varies from OEM to OEM. The exceptions to the cheating rule appear to be Motorola and -- you guessed it, Apple fanboys -- Apple.

Also optimization-free are Nexus devices, as those are stock Android, and AOSP does not contain the tweaks. This explains, the report said, why the Nexus 4 benchmarked poorly last year.

Here's what they said:
We started piecing this data together back in July, and even had conversations with both silicon vendors and OEMs about getting it to stop. With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization.

It's possible that older Motorola devices might've done the same thing, but none of the newer devices we have on hand exhibited the behavior. It’s a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung.
That being said, it's possible to work around the optimizations to get true results, but why should anyone have to. Benchmarks should give an honest comparison between devices.

The full report is quite illuminating, and should be checked out at AnandTech.

No comments: