Did Samsung just get caught gaming benchmarks?
A report by Anandtech, a site specializing in benchmarks and other performance-based analysis, uncovered a whopper on Tuesday: the site found code that apparently triggers the international version of the Samsung Galaxy S4 to run at a faster speed when certain benchmarks are performed.
Specifically, Samsung—which manufactures the phone itself as well as the 8-core Exynos 5 Octa chip within it—apparently programmed the chip to run at its full rated speed of 533MHz when in the presence of four benchmarks: AnTuTu, Linpack, Benchmark Pi, and Quadrant. Otherwise, the site claimed, Samsung programmed the Exynos 5 Octa to run at 480MHz during other times, even while playing games.
For its part, Samsung said early Wedneaday that it clocks down the chip to play games, to prevent the phone from overheating. "[A] maximum GPU frequency of 533MHz is applicable for running apps that are usually used in full-screen mode, such as the S Browser, Gallery, Camera, Video Player, and certain benchmarking apps, which also demand substantial performance," Samsung said on its blog. "The maximum GPU frequencies for the GALAXY S4 have been varied to provide optimal user experience for our customers, and were not intended to improve certain benchmark results."
TechHive tested the U.S. version of the Galaxy S4, which uses a Qualcomm APQ8064T chip, and found no such discrepancies.
The kicker, however, is a string of code that the site discovered, which the code identified as “Benchmark Booster.” In the presence of the benchmarks, the code instructed the processor to run at the higher 533MHz speed. The site also reported that the string appeared to be designed so it could be called by other benchmarks—running at the higher frequency in the presence of those other benchmarks in the future.
Why it matters
The upshot? If benchmarks are used to compare the performance of the Galaxy S4 against rival phones, Samsung could receive an unfair advantage. (TechHive this year stopped using synthetic benchmarks to evaluate phones.)
Unfortunately, the gaming of benchmarks is a shameful tradition within the PC industry. For instance, companies like ATI and Nvidia have been accused of altering their driver software to skew test results favorably. However, FutureMark, the organization in question, later retracted its allegations. Nevertheless, it’s a surprise, and a sad one, to see those tactics in use in the mobile space.
Anand Lal Shimpi and Brian Klug, the authors of the Anandtech piece, said as much.
”We’ve said for years now that the mobile revolution has/will mirror the PC industry, and thus it’s no surprise to see optimizations like this employed,” the two wrote. “Just because we’ve seen things like this happen in the past, however, doesn’t mean they should happen now.”
Do you buy Samsung's explanation? Or is the "Benchmark Booster" just that?
Additional reporting by Armando Rodriguez. This story was updated at 10:07 AM PT on July 31 with comments from Samsung's blog.
This story, "Did Samsung game its Galaxy S4 benchmarks with 'Benchmark Booster'?" was originally published by TechHive.