How to Benchmark Your Android Device

How powerful is your phone? The question would have been nonsensical a decade ago, but as we increasingly depend on smartphones and tablets for some of our everyday computing needs, paying more attention to their performance makes sense. Until recently, choosing a smartphone was mainly a matter of aesthetic taste and mobile OS preference. Today, however, most of the devices available prioritize their display and use similar slate or candy-bar designs. As a result, performance has become a much more prominent differentiator.

Unfortunately, many mobile device owners are falling into the same trap that desktop PC consumers did years ago: They look at a list of specifications, see that a smartphone’s processor is clocked higher or has more cores, and assume that the device is faster than one that has what appear to be lower specs. It’s as though the MHz myth of the 1990s is back to confuse buyers in a totally new market segment.

The standard SoC (system on a chip) powering one of today’s mobile devices, however, is advancing at a rapid pace. A device produced just a few months ago may have the same numbers of processor cores and be clocked at the same frequency as a brand new device, but its underlying architecture may be far less efficient. In addition, the two devices’ SoCs may be outfitted with different GPUs, different amounts of memory, and different versions of the mobile OS. Because of these hidden variables, a vendor’s list of specs doesn’t reveal the whole story. As is true with desktop and laptop PCs, establishing the relative performance of mobile devices requires testing.

Getting Started

Since Android’s ecosystem is far more diverse than its mobile OS rivals’, we’ll focus on testing an Android-based device here. A handful of the tests that we’ll be covering here are cross-platform or run within a Web browser, meaning that they’ll work on iOS or Windows Phones-based devices–or even on desktops and notebooks–as well.

In evaluating the performance of an Android-based smartphone or tablet (or any other modern smartphone or tablet), it’s best to think of the device as a tiny PC. When benchmarking a full-fledged desktop PC, we don’t declare that system faster or slower than another after running a single test that stresses a particular component. Instead, we run a host of different tests to gauge how well various individual components work, then we weigh the results, and finally we reach a judgment about overall performance. The same approach is appropriate for Android-based devices.

A number of benchmark tools are available online and in the Google Play market for testing device performance. We’ll discuss a handful of free tools here, but many other benchmarking tools are available, too. Some (like Vellamo from Qualcomm) are offered by hardware manufacturers; others are created by students. We recommend using established tools whose makers have demonstrated a capacity and willingness to issue updates; ultimately, however, you should tailor your benchmark suite to your unique usage habits. If Web browsing is your highest priority, use apps that test browser and JavaScript performance. If you spend most of your time on mobile devices playing games, be sure to run graphics/GPU-centric tests.

Benchmarking Procedures

As with PC performance testing, there are some standard procedures to follow when running benchmark tests on Android-based devices. Apps or utilities running in the background on an Android-based device can sap system resources and interfere with the benchmarks, resulting in inconsistent or misleading scores that don’t accurately reflect real-world performance. To eliminate this variable, your best course is to shut the background apps down. Many Android-based devices include built-in Task Managers that will turn off unnecessary background apps, thereby freeing up memory and processor resources. Restart your device, wait a few minutes to ensure that everything has loaded completely and then use the task manager to kill unneeded apps and maximize available memory. Wait a few moments longer, and then begin a fresh test.

Processor, Memory, Storage, and Overall Performance Tests

Linpack for Android (available at Google Play store, free)

Linpack for Android is a quick and easy-to-run benchmark that measures how fast an Android device can solve a dense N by N system of linear equations. Linpack is purely a CPU benchmark, and it can run in either single-thread mode (to assess the performance of a single CPU core) or multithread mode (to tax all of the available processing cores in a device). To start running the tests, you simply launch the app, choose the single-thread or multithread test, and then click the appropriate button in the main menu.

AnTuTu System Benchmark (available at Google Play store, free)

AnTuTu System Benchmark is an all-in-one benchmark designed to run tests on your device’s CPU, GPU, memory, and storage. The CPU benchmark evaluates both integer and floating-point performance. The GPU tests assess 2D and 3D performance. The memory test measures available memory bandwidth and latency. And the storage tests gauge the read and write speeds of a device’s flash memory. To run AnTuTu, swipe to the Test tab, select the tests that you’d like to run, and then click the Start button. When the tests are complete, the results will appear on the Scores tab.

CF-Bench (available at Google Play store, free)

CF-Bench is another all-in-one benchmark, created by respected Android developer Chainfire (hence the CF). Chainfire originally built the tool for his own use, but a couple of years ago he released it to the public. CF-Bench is a CPU, memory, and storage benchmark tool that can exploit the additional computing resources of multicore SoCs; and it produces consistent, repeatable scores, while testing both native and managed Java code performance. To run CF-Bench, launch the app and click the Full benchmark button. Results will appear at the bottom of the screen, along with a handy comparison to other popular devices.

Next: GPU/graphics, network, and browser performance

GPU and Graphics Performance

GLBenchmark (available at Google Play store, free)

GLBenchmark is an OpenGL ES 2.x benchmark composed of 33 different tests that evaluate a GPU’s ability to handle texture-based and direct lighting, bump, environment and radiance mapping, soft shadows, and vertex shader-based skinning, among other things. In addition to its diverse test suite, GLBenchmark offers a couple of “off-screen” tests that it renders at the same resolution (1280 by 720), regardless of the device being tested and regardless of any V-Sync limitations. Many mobile graphics benchmarks output results at right around 60 frames per second, because the devices’ screens are locked to 60Hz, which renders the results useless. To run GLBenchmark, you must first create an account; but once you’ve done that, all you have to do is launch the app, click the Performance Tests button, choose your tests, and click Start.

An3DBenchXL (available at Google Play store, free)

An3DBenchXL–a more taxing, updated version of the original AN3DBench–is a free, easy-to-use benchmarking tool based on an Android port of the free jPCT 3D engine/API. The app runs through a series of five tests that employ different rendering techniques with varying polygon counts and texture workloads to evaluate a GPU’s performance; then it spits out a series of results that you can compare against the results for other smartphones, tablets or even your PC and console. Such comparisons are possible because AN3DBenchXL reports the results of the five individual tests in frames per second (an overall score is generated as well) so you can see how your device stacks up against other gaming machines. An3DBenchXL couldn’t be easier to run, either: Just launch the app and then tap the screen to start. The results will appear after all of the tests have been completed. Make sure to follow the benchmarking procedures outlined above to ensure optimal performance.

Network Performance

Ookla Speedtest.net (available at Google Play store, free)

Ookla’s Speedtest.net app, an extension of the popular Speedtest.net website, is designed to evaluate the speed of a mobile device’s Internet connection. This is a great tool for illustrating the stark differences in performance that exist between 3G and 4G mobile networks and even Wi-Fi. To run Speedtest.net, launch the app, give it a few moments to pinpoint your location and choose an appropriate server, and then click the big Begin Test button. The test will measure and report your Download, Upload, and Ping times and will display a real-time graph that tracks connection consistency.

Browser Performance

Rightware BrowserMark (available to run on Rightware’s site)

As its name suggests, Rightware’s BrowserMark is a tool for measuring a browser’s JavaScript and HTML rendering performance. BrowserMark runs through an array of animations and JavaScript functions; at the end of the test it displays a final score, which you can use to compare the performance of different devices. To run BrowserMark, point your browser to http://browsermark.rightware.com/ and click the Start button. If you plan to compare the performance of different devices, you should use the same browser for all of your test runs. Performance varies greatly from browser to browser in BrowserMark, which makes this benchmark a good tool for seeing which browser runs best on your device.

SunSpider JavaScript Benchmark (available to run on WebKit’s site)

SunSpider is another browser-based benchmark. This test, however, strictly tests the performance of JavaScript functions, which are used widely across the Web, excluding Document Object Model (DOM) and other browser APIs. To run SunSpider, point your browser to www.webkit.org/perf/sunspider/sunspider.html and click the Start SunSpider 0.9.1 now link. The site will conduct a number of tests, and then it will display a final result (as well as the individual test scores). On SunSpider JavaScript Benchmark, lower scores indicate better performance.

Next: Benchmark results

Benchmark Results

This lengthy discussion of Android benchmarks would seem discouragingly abstract without some reference data, so we ran through all of the tests cited in this article on a popular Android-based device, the Samsung Galaxy S II i777 smartphone. The Galaxy S II i777 comes configured with a 1.2GHz dual-core Exnyos C210 SoC, 1GB of RAM, 16GB of internal flash storage, and a 4.3-inch screen that has a resolution of 480 by 800. The Galaxy S II i777 is also designed to use AT&T’s pseudo-4G HSPA+ wireless network (non-LTE), and at this writing it runs Android Gingerbread 2.3.6, though an Android 4.0 Ice Cream Sandwich update–which should significantly impact performance–is supposedly on the way.

Until the arrival of the next wave of quad-core devices, or devices based on newer ARM or Intel architectures, the Galaxy S II i777 qualifies as a high-performing smartphone. If you run these tests on your Android-based device and find that it delivers lower performance on a particular benchmark, don’t fret: Most currently available dual-core devices won’t outperform the Galaxy S II. If you get scores much better than the ones tabulated below, rest assured that you have a speedy device on your hands. Congratulations, and happy testing!

Processor
Linpack for Android v1.2.8
Single thread	57.30 MFlops
Multithread	76.18 MFlops
AnTuTu System Benchmark 2.7.3
CPU integer	1689
CPU floating-point	1110
Database IO	420
RAM	891
SD Card write	125 (12.5 MBps)
SD Card read	203 (>50 MBps)
CF-Bench (native scores)
MIPS	1021
MSFlops	562
Memory read	3117
Memory write	1594
Disk read	552
Disk Write	511
Overall score	6432
GPU
GLBenchmark v2.1.4
GLBenchmark 2.1 Egypt	44 fps
GLBenchmark 2.1 Pro	58 fps
An3DBenchXL
Double Dragon	18.91 fps
Flower Power	38.07 fps
Ninjas’ Garden	33.36 fps
Emperor’s New Clothes	59.4 fps
Magic Island	24.5 fps
Total score	30,171
Network
Ookla Speed Test	Wi-Fi	4G
Ping	17 ms	70 ms
Download	20,607 kbps	6327 kbps
Upload	8254 kbps	1059 kbps
Browser
Rightware BrowserMark
Total (Android browser)	48,939
Total (Dolphin HD browser)	54,853
SunSpider JavaScript Benchmark v0.9.1
Total	3497.3 ms
*Test device: Samsung Galaxy S II model i777