How to Benchmark
Remember a few rules of thumb:
- Prep for benchmarking (including cleaning out the system).
- Let the system settle down for a bit.
- Understand the benchmark.
- Check for repeatable results.
- Read the results carefully.
- Don't draw conclusions that are too broad.
- Document, document, document!
Prep for Benchmarking
Before you fire up a benchmarking app and start running performance tests, you need to prepare the PC. In a perfect world, you want repeatable results, but the architecture of modern operating systems sometimes conflicts with that laudable goal.
Ideally, you want a clean system for running performance tests. But Windows runs a lot of stuff in the background. You should leave normal Windows services running; usually they don't have a major impact on benchmark results (unless something like a scheduled defrag or backup is occurring). However, third-party software often installs services and tray applications that use up CPU cycles, ping the network, or occasionally write to storage.
You can review your machine's background items by running msconfig.exe: Open the Start menu, type Run, and then type msconfig in the Run box.
In the example shown here, I'd uncheck Microsoft Office, Adobe Updater, all the Logitech crud, the Acronis backup helper apps, and other similar stuff. If this were a new PC on which I'd want to run baseline performance tests, I'd probably uninstall any antivirus or security programs as well (remembering to reinstall them when the machine was back in actual use).
One other major headache is making sure that all needed updates--including Windows patches--are installed. After you install the latest Windows fixes, it's best to disable the Windows Update service. (You'll find the control panel link to change Windows Update settings under the 'System and Security' heading in Windows 7.)
Make sure all your drivers are up-to-date as well--particularly the driver for your graphics card, but also the audio driver.
Once you have a clean, updated system, you're almost ready to start.
Let the System Settle
After you first boot the system, Windows does a lot of stuff--loading services and drivers, defragging the hard drive, reordering applications on the drive, and more. It's a good idea to let Windows settle for a bit.
The amount of time varies depending on the system's age. A brand-new PC with a just-installed Windows configuration may take several hours to settle down. In fact, if you really want the best possible benchmark numbers, it's wise to install all the benchmarking software plus any commonly used applications, and use your PC for several weeks first. It takes a week or two for Windows to optimize the disk layout and memory caches appropriately for your system usage. Admittedly, the differences will be minor-- maybe a percent or two. But letting the system idle for 15 minutes or so makes a bigger difference; running a benchmark on a just-booted system will practically guarantee variable results.
Understand the Benchmark
When you install a test, you might be interested in just firing it up and running it at the default settings. But unless you understand the context and goals of the test, the numbers it produces won't mean much to you.
Take particular care if you're running game benchmarks. Some games will reset their detail settings if you install a new graphics card--or even install an updated driver. If you run the test with different detail settings, it's impossible to get a valid comparison.
If the benchmark has documentation, read that carefully. You may discover some inherent issues that give one type of component an advantage in the benchmark that isn't always reflected in the real world. One example of this is the 'PPU Enabled' setting in 3DMark Vantage. (PPU stands for "physics processing unit.") Leaving PPU enabled boosts scores on Nvidia-based cards only. Is that fair? Maybe it reflects what you'd see in a game that uses Nvidia's proprietary PhysX physics engine, but not so much for most games.
Knowing the ins and outs of your performance tests is vital to ensuring repeatable results. Which brings us to our next topic...
Check for Repeatable Results
In trying out a new benchmark, run the test several times on the first go-around. Can you get results within a 2 to 4 percent window? If the differences exceed 5 percent, either something is wrong with the benchmark, or your system isn't quite settled or prepped properly.
Or, perhaps the benchmark settings have changed. Altering default settings can bite you if you're not careful. For example, when I run the Unigine Heaven benchmark, I change the default settings, but the test resets itself every time it runs. So I have to take special care to ensure that I reenable the settings.
Benchmarks that allow you to create batch files with custom command-line settings are terrific, because you can always be sure that the settings are what you want, even if the benchmark resets itself upon exit.
The flip side of the coin is that if you consistently get results within 3 percent of one another, that's good enough. People tend to agonize over a sub-1 percent difference in benchmark results. Even if that sub-1 percent number is repeatable, it's generally irrelevant in terms of real-world performance.
Remember, if you want repeatable results, make sure that all things are equal, except the one thing you're changing. For instance, don't test different hard drives in different systems--always use the same system for testing the same type of component. That gives you a frame of reference for your results. If you're changing multiple items of hardware at the same time, it's hard to figure out exactly which one was responsible for any change in performance.
Read the Results Carefully
Some performance tests take special care to present results in a format that's easy to understand. Others seem to go out of their way to confuse you.
For example, here's a results presentation from a popular game benchmark:
Average FPS: 39
Minimum FPS: 13
Maximum FPS: 79
It's easy to look at this result and write down "79." Whoa, that new graphics card really is fast. But if you've been recording the average frame rate, such a quick glance will mean you've just skewed the results unfairly--and completely unintentionally. So read the results with care.
Don't Draw Overly Broad Conclusions
Let's say you run a popular CPU benchmark--something like Prime95--and you find out that your machine's score is relatively low compared with those of other processors. This depresses you somehow, because your system will now be forever inadequate in all applications.
Actually, that may not be true. For example, your PC may run some games just fine. Or, if you have ample DRAM installed, your system's performance in photo-editing applications will be pretty good.
It's easy to draw a sweeping conclusion from a couple of tests. Resist the urge and remember: Run the benchmarks that matter for your own system usage pattern.
Document, Document, Document!
You've just installed more memory that's also faster than your previous kind. Your spiffy new PC now has 4GB of DDR3 running at 1600MHz, up from 2GB at 1066MHz. You run several performance tests to check for any improvement. After looking at the numbers, you think the numbers are better.
You did record your earlier results, right?
I keep master spreadsheets of most of my performance tests, usually by component type. I can fire up my graphics-card spreadsheet and look at results going back several years. Such information is really useful, because we want to believe that newer hardware runs applications faster, but that isn't always true. Unless you document past results, any actual gain is hard to determine.
Go Forth and Benchmark
Now that you understand some of the basics of performance testing, download a few benchmarks and give it a try. Whether you want to compare different components, troubleshoot a performance issue, or see if your system tweaks have had any effect, benchmarking will help you better understand your PC and how it performs.