How to Benchmark Your Browser for HTML 5

How to Benchmark Your Browser for HTML 5
Contemporary browsers are much more than just a window into the World Wide Web: Browser developers have turned the software into sophisticated application platforms in their own right. But browsers are not the same as hardware platforms--rather, they function as virtual environments accessible from a variety of platforms. For example, you can have Google’s Chrome browser on Windows, Mac OS, Linux, and Android devices.

The current set of Web browsers--Chrome, Firefox, Opera, Internet Explorer, and Safari--support assorted standards, including HTML 5, the latest version of HyperText Markup Language. HTML 5 is an ambitious extension of HTML, incorporating an array of features. The standard is still in flux, however, and the World Wide Web Consortium hasn’t finalized it yet. Some of the important new features in HTML 5 include canvas rendering, tighter integration of SVG (scalable vector graphics), and video and audio tags. These new elements are specifically designed to make it easier for Web developers to present and manage multimedia content. What HTML 5 currently doesn’t have is a built-in standard way to handle 3D graphics.

Whenever new, competing platforms emerge, it’s natural to try to compare their performance. After all, users want the most robust and most responsive environment for running their applications, whether those apps are for productivity, entertainment, or education. The problem is that Web applications themselves are in a state of flux, as is the state of benchmarking Web browsers.

What I’m going to cover in this article won't tell you what the fastest browser is, nor what the best hardware might be for those browsers--that will come later. Today, I'll dive into the complexities of benchmarking browsers, look at a few examples of benchmarks, and help you understand where performance testing currently stands when it comes to these new virtual platforms. By the end of this article, you will know how to benchmark and optimize your browser for HTML 5 applications.

What Are We Benchmarking?

Performance testing can sometimes be a bit of a black art, and with Web browsers the situation is even worse. Part of the problem is the relative immaturity of the platform--HTML 5 is still an emerging standard, after all. As a result, applications use only some pieces of the standard, if any. The other main problem is how existing benchmarks work: For the most part, they test somewhat different things, so you need to run several to get a clear idea of how well your browser will perform. For example, Futuremark’s Peacekeeper browser benchmark bills itself as mainly a JavaScript benchmark, but it makes use of HTML 5 canvas and video in its test.

In addition, the same browser may behave differently on different operating systems. The Mac OS incarnation of Apple’s Safari, for instance, runs better than its Windows kin does (not a big surprise).

As you begin to consider benchmarking, you need to ask yourself a few key questions. Which platform? Is your intent to benchmark browser performance, or hardware? Are you running on a desktop system, a laptop, a tablet, or a phone? All of these questions will help to determine the type of benchmarks you’ll want to run. I’ll be focusing on desktop browser performance here, but bear in mind that performance in, say, Chrome running on an Android tablet with a power-efficient GPU and ARM processor will be different than on a desktop system with a discrete GPU.

What is cool is that we now have a way of benchmarking across multiple hardware platforms, albeit with some caution. Since Chrome and Firefox run across many different operating systems, you can use them to test performance across those platforms. Of course, the way code is compiled and built will differ, and some custom code is necessary for a browser to run on a particular OS or hardware platform, which can affect performance. But it’s a good first step.

I’ll be taking a look at several scenarios, all using desktop browsers. First, I’ll examine performance across three browsers running under the Windows 8 Consumer Preview. I chose Windows 8 mostly because I was curious about the claimed performance increases of Internet Explorer 10 over IE 9.

Access HTML 5 test pages such as Asteroids to benchmark how well your browser can handle HTML 5.

In addition to checking out browsers, I also swapped graphics cards. For the first run, I used an AMD Radeon HD 6970; for the second, an Nvidia GTX 570. Both were reference boards, so they ran at default clock speeds.

Finally, just for comparison, I tried the same benchmarks on a fairly typical Windows 7 laptop.

Next Page: A Look at the Benchmarks

The Benchmarks

A large number of HTML 5 and JavaScript performance tests exist; I selected three that looked pretty good. Of the three, only one--Futuremark’s Peacekeeper--behaves like the standard benchmarks designed to assess desktop PC performance. Peacekeeper runs a range of tests, including Web page rendering, HTML 5 with WebGL, HTML 5 canvas, and HTML 5 video, and it generates a single score at the end.

Futuremark's Peacekeeper tests a wide range of Web rendering capabilities, including various HTML 5 features.

Asteroids looks 3D, but it’s really a 2D HTML 5 canvas drawing test. The artwork looks slick, and it’s based on an actual game developed in HTML 5. Like Peacekeeper, it spits out a single score, but it also generates an average-frame-rate number.

The third test is the Impact HTML5 Benchmark. This one is also based on an actual HTML 5 game, and the author admits on his blog that Impact “tests one very specific use case: smooth running games rendered with the element.”

HTML5 Benchmark runs through a side-scrolling shooter game level, purely testing performance in this style of game using HTML 5.

Benchmarking Your Browser

One important note: For benchmarking, you need a robust broadband connection. I’m not talking about raw bandwidth, but rather reliability. The last thing you want during a benchmark run is to have your system burp due to a DNS error or some other network-related problem.

As with any performance testing, you’ll want a clean system. In this case, however, that means something a little different than the norm: In hardware benchmarking, of course, you want a clean install of the OS, but with Web browser testing you also need to make sure that the browser itself is clean. This process involves tidying up several different aspects of your browser.

  • Uninstall your existing version and do a fresh install. In today’s world of automatically updated browsers, this step isn’t completely necessary, but starting fresh is still the best approach.
  • When you install, make sure not to install additional toolbars inadvertently. Sometimes browser installs--particularly if they don’t come from the original source--will carry assorted toolbars as options. Typically you have to opt out to ensure that the installer doesn't add these toolbars to the browser.
  • Purge all add-ins and plug-ins. It’s not good enough to just check the browser’s plug-in page in the options settings. If you’re running on Windows, you should also look at the Windows Add/Remove Programs control panel, since some browser add-ins need to be uninstalled the same way normal applications do.
  • Confirm that your network and graphics drivers are current.

When you run the benchmark, have only the one tab open (unless the benchmark itself opens additional tabs). Most of these tests don’t have performance-sucking Flash advertisements or background animations running, but the sites in your other tabs might.

Next Page: Browser Performance Sample Results

Windows 8 Browser Performance

Before I dive into the scores, let me first run down the Windows 8 system I used:

  • Intel Core i7-3820 running at 3.6GHz
  • 16GB DDR3-1333 RAM
  • Gigabyte X79-UD7 motherboard
  • Intel 250GB solid-state drive
  • AMD Radeon HD 6970 reference graphics board with Windows 8 drivers installed
  • Windows 8 Consumer Preview

Let’s take a look at the performance chart I generated using the benchmarks mentioned on the previous page.

As of version 17, Chrome seems faster than the version of IE 10 shipped with the Windows 8 Consumer Preview, using these particular benchmarks. The differences are significant, but not huge. On the other hand, Firefox 11 trails pretty badly in all cases.

Hardware Change

Okay, so now we have an idea of how three different browsers perform in the Windows 8 Consumer Preview. What happens if we vary the hardware? In this case, I removed the Radeon HD 6970 and dropped in an Nvidia GTX 570 graphics card; the two boards are in roughly the same price category. Since modern Web browsers now take advantage of the GPU, I expected to see some differences with different graphics hardware.

As you can see, the hardware does have an effect on how well HTML 5 benchmarks will run in a browser. According to this set of tests, the Nvidia GTX 570 looks to be just a little faster than the Radeon HD 6970 in many browser applications, particularly in Chrome. The differences are smaller than those I noted earlier, but they can still help you understand which browser is best for running HTML 5 applications on your particular system.

Typical Windows 7 Laptop

I also assessed browser performance in Windows 7, and the results were considerably different. I ran the same tests mentioned above on an HP Envy 14 Spectre, a fairly typical modern laptop. While the chassis looks impressive, the internal components are standard fare, including an Intel Core i5 CPU (1.6GHz base clock, with a 2.3GHz max turbo frequency), 4GB of DDR3 RAM, and an integrated Intel HD Graphics 3000 chipset. Here's what HTML 5 performance looked like across three different browsers.

As you can see, differences in platform certainly can affect your browser's performance results. In my tests, Microsoft Internet Explorer 9 performed poorly while executing the Peacekeeper test in Windows 7, but managed to hang in there on the dedicated gaming benchmarks. Firefox running in Windows 7 made a big comeback in the Impact performance tests, but still lagged on the others. Perhaps the biggest surprise is that Google's Chrome didn't perform nearly as well on the HTML 5 tests (comparatively speaking) in Windows 7 as it did in Windows 8.

Your Turn

As with any platform tests, your results will differ from mine. Performance across assorted browsers or hardware platforms will vary depending on the operating system, the underlying hardware, plug-ins, and other variables, some of which will have a substantial impact. Also, browsers are evolving at a blistering pace, so today’s winner might be tomorrow’s also-ran.

So take these particular results with a grain of salt, and keep in mind that I ran a small subset of tests on one type of hardware. You might give these benchmarks a whirl on your favorite browser, and then tweak your hardware and software accordingly. Now that you know how to conduct proper benchmark tests, it's easy to see for yourself how much of a difference new hardware (such as a graphics card) or software (like Windows 8) can make.

Subscribe to the Daily Downloads Newsletter

Comments