But what about Geek Bench 3?
The bulk of these tests would indicate the A9X is a really fast ARM chip, at the top of the mountain—for ARM-based tablets. If you still want laptop-like performance with a full-service OS though, you’ll need a full Core-class chip
That’s not what you’d think if you peeped at Geek Bench 3. Per core, remember, it’s a CPU test that shows the iPad Pro to be every bit as fast as a current Core i5 Skylake CPU.
Some have claimed the problem is how Geek Bench 3 weights its results. SHA2 encryption, for example, is overly represented for CPU performance. Given the hardware acceleration in the A9X, it’s showing Apple’s chip to be far faster than that it actually is.
Geek Bench 3 lets you view sub scores, and here’s how SHA2 performance played out across various devices, including a desktop 3.4GHz Core i7-2600 and a water-cooled 4GHz Core i7-4790K chip. Because it’s single-core performance, the Hyper-Threading and additional cores don’t cloud the issue. The results are rather interesting.
If you believe Geek Bench 3’s SHA2 numbers, the A9X in the iPad Pro and Nvidia’s Tegra K1 are actually faster than all of Intel’s current mobile dual-cores in SHA2 encryption performance.
Both are also faster than a desktop Sandy Bridge chip. To put that in perspective, the SHA2 performance of the Tegra K1 and A9X aren’t too far behind that of an 88-watt 4GHz Core i7-4790K chip.
Geek Bench 3 is hardly without controversy either. Often cited by its detractors is a post by calm sage Linus Torvalds from a forum in RealWorldTech.com, where he went Kanye West on Geek Bench 3.
”Wilco, Geek Bench has apparently replaced dhrystone as your favourite useless benchmark,” Torvalds wrote. ”Geekbench is SH*T.”
”It actually seems to have gotten worse with version 3, which you should be aware of,” Torvalds wrote. “So basically a quarter to a third of the ‘integer’ workloads are just utter BS. They are not comparable across architectures due to the crypto units, and even within one architecture the numbers just don’t mean much of anything. And quite frankly, it’s not even just the crypto ones. Looking at the other GB3 ‘benchmarks,’ they are mainly small kernels: not really much different from dhrystone. I suspect most of them have a code footprint that basically fits in a L1 cache.”
Geek Bench’s side of the story
To get Geek Bench’s side of the story, I spoke with John Poole, one of the primary developers behind the test. Poole said he understands the controversy and has taken it to heart, but he also disagrees with Torvalds.
“We have a lot of respect for him,” Pool said. “I think he’s wrong in this case.”
Torvalds argues against the value of small code loops in measuring performance, but Poole said the future is mostly about smaller loops. Poole said moving a window around a screen or opening a window is mostly a solved problem for CPUs.
Poole said they’ve been very transparent with what the test measures and have provided extensive documentation as well. In order to measure the chip performance, Geek Bench tries to execute the same code on every platform, Poole said.
Poole claims the question of whether the A9X is faster than, say, a Core m3 is beside the point. Today, the software that you can run on a laptop just isn’t available on the iPad Pro, rendering Apple’s productivity tablet mostly a curiosity until software changes that.
Still Poole said he does wonder how the A9X would run OS X. I do too, as it would make comparisons far easier to measure.
To be fair to Geek Bench, benchmarks developers are often accused of serving internal political needs. BAPCo, the software and hardware consortium, has its roots in the PC industry and some feel it’s too PC-centric. Even among PC vendors, there was strife when AMD quit the group and accused BAPCo of being too closely aligned with Intel.
BAPCo officials always say their intentions are to create benchmarks that offer insight to the public. If the test is cooked to favor Intel or x86, it certainly didn’t show when the new version was released earlier this year and Intel’s older Atom chips were kicked down the stairs by Apple’s and Nvidia’s CPUs.
I spoke with BAPCo’s John Peterson about the philosophical differences between the two.
”TabletMark utilizes a wide variety of APIs provided by each platform to represent productivity/media app performance and battery life. The workloads are implemented in a way that’s meant to reflect the implementation choices app developers would make for each of the platforms,” Peterson said. “GeekBench generally utilizes its own libraries to perform tasks, while TabletMark utilizes the available platform APIs to perform equivalent functionality in a more platform-tailored way.”
Peterson also said relying on browser-based benchmarks, which many testers have done, is inherently problematic, as you’re limited to what hardware is exposed to the browser.
What about editing three 4K streams?
One of Apple’s most impressive claims is the ability to “edit” three simultaneous 4K movie streams—something that is no fun on a desktop and probably out of reach of most laptops.
To test that, I decided to throw the platform a fastball. I grabbed several 4K RAW video files shot on a Red digital cinema camera.
On the Core i7 Surface Book, I installed Premiere Pro CC, created a project, added several of the R3D files to the timeline, and tried to play it back. Without doing a render, it wasn’t going to happen at full-res without major hitching—and this with the assistance of the GeForce GPU. Once rendered out, editing and scrubbing through the timeline was possible.
On the iPad Pro I tried to open up the same R3D file in iMovie with no luck. I’m pretty sure the issue is iMovie’s inability to read the raw R3D files. I also copied over a 4K-resolution .MOV file to the device but had no luck opening it in iMovie either. I’m not saying Apple is wrong, but I had no luck trying to do what Apple says the iPad Pro can do.
So what does this all mean?
For one thing, I don’t think you can look at just a couple of numbers from one or two benchmarks and make a conclusion. That is, unless you’re looking to bend the truth to fit a pre-conceived agenda. That’s called benchmarketing, not benchmarking and there’s a difference.
The truth is, most of the testing I’ve run shows the iPad Pro isn’t faster than a current or even two-year old Core-class Intel CPU. (Atom, now that’s another story.)
It’s just not.
But it’s still one hell of a chip.
I tried to heat up the A9X to check for performance throttling by repeatedly running 3DMark and simply gave up. It just does not get hot. I can’t say the same for the admittedly smaller (and harder-to-cool) Google Nexus 9, which gets hot just browsing the web. Watching the same 4K movie file (that I couldn’t actually edit) on the iPad Pro was buttery-smooth even playing in a background window. It’s a very impressive tablet.
It still won’t replace my Surface Pro 3 nor my laptop because it’s not up to desktop-grade functionality. That’s not the A9X’s fault. It’s because the OS and apps aren’t up to the “pro” requirements for multi-tasking, nor the precision control of a mouse and keyboard experience.
If Intel’s Atom X7 can run a full-service OS, the superior A9X theoretically could, too. That’s a win for Apple in the long run, and it should be a wake-up call for Intel.