Virus Stoppers

How We Test Antivirus Software

The PCW Rating for stand-alone antivirus software is based on separately calculated scores in each of our usual four categories: Performance, Specifications, Design, and Price.

The Performance score, heavily weighted in the PCW Rating, measures not only system speed tests but also the critically important malware detection and disinfection tests. The detection tests are heavily weighted within the performance score.

To perform detection and disinfection tests, we partner with AV-Test.org, a security research company in Germany. AV-Test.org puts programs through a rigorous analysis; its overall malware detection test pits each app against an almost 900,000-sample "zoo" of viruses, Trojan horses, back doors, and other malware types. Some of these samples are commonly used in Internet attacks; others are far less well known and may have been used in small, targeted attacks. For these on-demand zoo detection tests, the antivirus programs are set at their best detection settings.

Outbreak response time tests rate how quickly each antivirus company delivers the malware signatures used in these detection tests, as measured by AV-Test.org. The reported time ranges are based on when the first company begins releasing signatures, since determining when a given piece of new malware first surfaces in an attack can be difficult or nearly impossible.

To perform proactive tests that simulate how well the programs can detect unknown malware, AV-Test.org scans a set of new malware with each app using one- and two-month-old signature files. The detection tests also examine how well each program handles different document types, such as whether it can find malware hidden within various types of archived files.

The disinfection tests measure how well an antivirus app can detect and then clean an existing infection that has installed itself on a PC. AV-Test.org checks whether the antivirus software has removed malware files, changes to the Hosts file, and Registry changes. We consider cleanup of Registry entries the least important of the three areas, and weight it less heavily.

To round out the Performance score, we test the antivirus software's impact on a PC's speed. For that we use portions of PC World's WorldBench 6 Beta 2 benchmarking application. These tests measure how long a computer takes to run a set of automated tasks with a variety of programs, including Firefox, Microsoft Office, WinZip, and other apps. We run WorldBench 6 Beta 2 on a test PC multiple times with the antivirus software installed and then without. By comparing the results with and without the software installed, we can calculate how much system drag or slowdown each antivirus program adds.

The Specifications score gauges each program's basic feature set. While it's important, it's not nearly as heavily weighted as the Performance score. We look for things such as whether the program scans e-mail and Web traffic to catch attacks before they hit the hard drive and whether the program can manually delete a file via a right-click menu option in Windows Explorer. Also in this section, we research the type of support (phone and e-mail) a program offers and any costs associated with it.

Our Design score evaluates each program's interface and ease of use. If a program makes it easy to find and understand program settings, looks good, and installs with default options appropriate for the average user, it scores well here.

Finally, we rate Price. For the sake of consistency, we chose the download price rather than the price of a retail box where there was a difference. We considered the purchase price of a license for one computer for one year (except for products whose lowest price covers multiple computers), as well as the second-year renewal cost. Most companies offer different rates for multiple computers and/or multiple years, so check your options for the best deal for your situation.

You may wonder why our final scores for antivirus and other product ratings often bunch up (our three top-ranked products in "Virus Fighters" earned final scores of 85, 84, and 84, for example). In part, this result can be due to several products having similar (or in the case of the story just mentioned, nearly identical) performance in one heavily weighted area such as malware detection. Also, antivirus programs can have very similar feature sets.

For these reasons, it's important to look at each product's Test Report for its unique performance and feature characteristics. Two products with similar PCW Ratings may have significantly different results in certain individual performance tests and feature areas.

Ryan Naraine is a freelance technology journalist specializing in Internet and computer security issues. Erik Larkin is an associate editor for PC World.