How We Test

PC World does two major forms of testing of all products under consideration for review: hands-on testing by editors and writers who are experts in the product category, and formal lab testing by the PC World Test Center.

In the near future we will publish all of the methodology we use for our hands-on testing, which, like lab testing, follows rigorous guidelines to ensure fair and impartial analysis.

Following is a synopsis of the tests that the PC World Test Center performs for major product categories. Other major categories will be added soon, including our procedures for testing televisions.

Audio Players

All quantitative measurements of audio players are made with an Audio Precision ATS-2 audio analyzer with control software version 1.40. We simulate the use of typical earbud-style headphones by driving a 30-ohm resistive load. For each player we measure the following:

• Maximum output level: We determine the loudest volume the audio player can reach while playing a 997-Hz tone without exceeding 1 percent distortion.
• Distortion: We play a 997-Hz tone and adjust the player's volume control until the output is as close to -20 dBV as possible. We then measure the distortion and noise (TDH+N) in a 20-Hz to 20-kHz bandwidth.
• Signal-to-noise ratio: We measure how much noise each audio player makes relative to a -20 dBV output level while playing a silent MP3 file with dither. A higher signal-to-noise ratio is better.
• Cross talk: We play a 9.997-kHz tone in the left channel while measuring the right channel, and vice versa. We then measure how much energy leaks into the adjacent channel.
• Frequency response: We play an MP3 file that contains 30 tones from 20 Hz to 20 kHz. We report the absolute deviation between the lowest-level tone and the highest-level tone.

Backup Software

To assess each program's performance and its ability to back up open files, we back up a 2.5GB Windows installation to both a mapped network drive and an external hard drive. We also test the application's ability to span optical media by backing up 6GB uncompressed to rewritable DVD. To test claims of tape support, we perform a minimal 100MB backup to an Exabyte VXA-2 tape drive.

Camcorders

To gauge video quality, we shoot a complex scene with multiple moving elements, including a train set and toy Ferris wheel. We judge the quality of the audio played through a set of 2.1 stereo speakers. We shoot at both daylight and low-light settings, using the camera's highest-quality video and audio settings. A panel of judges reviews the recorded scenes from up to four different camcorders side by side on identical, calibrated television sets, and assigns video and audio scores; we then average those scores.

To gauge still-image quality, we take a series of shots at the camera's highest resolution. We photograph a complex still life and a target resolution chart to see how well each camera captures details and reproduces color. A panel of judges reviews printed photos and assigns image-quality scores; we then average those scores.

Cameras

Point-and-shoot models: We test all cameras with their included memory card. If no cards are provided, we use a Kingston brand memory card. To gauge picture quality, we take a series of shots, with and without flash, at the camera's highest resolution. We photograph a complex still life, a target resolution chart, and a mannequin to see how well each camera captures details and subtle coloring such as skin tones. A panel of judges reviews on-screen and printed photos and assigns image-quality scores; we then average those scores.

Advanced models and single-lens reflex models: We test all cameras with their included memory card. If no cards are provided, we use a Kingston brand memory card. To gauge picture quality, we take a series of shots, with and without flash, at the camera's highest resolution. We photograph a complex still life and a mannequin using automatic settings in Program/Full-Auto Mode to see how well each camera captures subtle color and exposure under its default settings. We then photograph the same still life and a resolution moiré chart with semiautomatic settings using aperture priority, custom white balance, and exposure bracketing. We pick the best shots of each of those two subjects for judging. We also test the camera's capability for minimizing noise using a range of ISO settings. We review the on-screen and printed photos and assign image-quality scores.

The image-quality rating of the camera is based on five categories: exposure, color, sharpness, distortion, and overall.

Cell Phones

To test for battery life talk time, we set all cell phones to vendor default settings. We don't change any of the power-saving settings, but we do maximize the volume setting.

We use a PC to play a looped MP3 of an audio book at a normal conversational volume. We position the PC speakers around a telephone hosting a conference call. We dial in the test units to this conference call and monitor the call with a software utility. If a phone disconnects, we determine if it simply lost its signal or if the battery has died. If the phone's battery isn't dead, we dial it back in to the conference call. We capped our tests at 10 hours. After completing two days of testing, we then average the results.

Desktops and Notebooks

We test or retest all desktop and notebook systems under our WorldBench 5 suite, and we put them through graphics testing as well. Notebooks also undergo battery-life tests.

WorldBench 5 is an applications-based benchmark that runs various tests using 12 applications selected for their mass-market appeal, stability, market share, and variety. PC World has always believed that application-based benchmarking is a better way to quantify computer performance than esoteric, synthetic methods. WorldBench 5 runs--albeit in an extremely compressed way--many of the same tasks that average users perform on their PCs every day.

WorldBench 5 uses the following applications:

• ACD Systems ACDSee PowerPack 5.0
• Adobe Photoshop 7.0.1
• Adobe Premiere 6.5
• Ahead Software Nero Express 6.0.0.3
• Discreet 3ds max 5.1 (DirectX)
• Discreet 3ds max 5.1 (OpenGL)
• Microsoft Office XP with SP2
• Microsoft Windows Media Encoder 9.0
• Mozilla 1.4
• Musicmatch Jukebox 7.10
• Roxio VideoWave Movie Creator 1.5
• WinZip Computing WinZip 8.1

For further information on WorldBench, 5 visit our WorldBench 5 page.

Starting with the June 2007 issue, PC World will include WorldBench 6 Beta 2 scores in tested projects until the final version of WorldBench 6 is released. Because of the updated tests and baseline system, scores from WorldBench 6 Beta 2 cannot be compared with scores from previous versions of WorldBench. For more information, see the WorldBench 6 Beta 2 FAQ.

Graphics tests measure the frames per second achieved for two popular games played at a variety of resolutions. The games were chosen to run on the widest variety of systems.

Battery tests measure battery life by repeating a 1-hour cycle of tasks until the battery is exhausted: typing and running a search-and-replace operation in Microsoft Word; typing and recalculating a worksheet in Microsoft Excel; doing a criteria search in Paradox; and playing Solitaire.

Document Scanners

We perform all scanning tests at 300 dpi. To determine the black-and-white scanning speed, we use three documents: a 30-page, double-sided document containing text and tables; a 15-page, one-sided document containing text, tables, and a few photographs; and a 2-page, single-sided application form. We scan each document as a TIFF file at 1 bit.

We determine color-graphics scanning speeds by using two documents scanned at 300 dpi and 24 bit: a magazine cover (scanned to TIFF), and a 2-page, double-sided brochure containing photos and text (scanned to PDF).

Among scanners tested for our April 2007 issue, however, was one exception: The driver for the Fujitsu ScanSnap s500 does not directly support TIFF, so we saved its multipage documents in PDF and its magazine cover in JPG format.

Graphics Boards

We test frame rates on a variety of popular games. Tests run at 1024 by 768 and 1600 by 1200 resolution, with and without 4X antialiasing and 8X anisotropic filtering enabled. You can find a complete list of the games used in the tests on the Test Report page of any graphics board review.

Hard Drives

To assess performance of external hard drives, we time how long it takes to copy and read 3.1GB of data (first as a number of folders and files, and then as one large .zip file) between our reference test bed's hard drive and the external drive we're evaluating. We also time how long it takes to perform a virus scan on 14.1GB of data stored on the external drive, and we search the entire contents of the external drive for files containing a certain text string.

To test internal hard drives, we copy 3.1GB of data (first as a number of folders and files, and then as one large .zip file) from one location on the drive to another; time how long it takes to scan a folder containing 6.2GB of data and program files for viruses; and time how long it takes to do a find-file operation on the entire drive in Windows XP.

For both external and internal drives, we single out the drive's results on the ACDSee PowerPack 5.0, Nero Express 6.0.0.3, and WinZip 8.1 tests contained in the release version of WorldBench 5. We carry out all tests on two identical test beds with a 3.6-GHz Pentium 4 processor and Windows XP Home Edition.

Monitors

A panel of 12 judges rates how well each monitor displays 11 text and graphics images at native resolution. For 17-inch and 19-inch monitors, native resolution is 1280 by 1024; for 23-inch and 24-inch wide-screen monitors, native resolution is 1920 by 1200. Some of the test images are from DisplayMate for Windows.

Optical Drives

We test optical drives under Windows XP Home Edition on a PC with a 1.67-GHz Athlon XP 2000+ CPU and 512MB of SDRAM, using the drive's bundled disc-burning software and media supplied by the vendor or by Verbatim. We test the drives with the format specified by the vendor at review time. Some vendors may change drive suppliers without altering their model designations; our results reflect the configuration tested at the time of our review. Write speed is the time (in minutes:seconds) required to write 7.9GB to double-layer DVD+R DL media, 4.35GB to single-layer DVD+R media, and 694MB to CD-R media, and to format a blank rewritable DVD+RW and packet-write 3.4GB to that disc.

Printers

We test printers for the number of pages per minute they can produce for text, and when appropriate, the pages per minute they can produce for color graphics and photos. We also assess them on text, line-art, color-graphics, grayscale, and color-photo quality.

We test for speed using a ten-page text document; a three-page text document that contains images and different-size fonts; a black-and-white photo using the black cartridge; a two-page spreadsheet, a three-page presentation, and an image using color cartridges; and an image with the printer set for best color-photo quality.

Multifunction printers receive additional tests for scanning speed and quality, as well as for copying speed and quality.