Test Results: Does SP1 Fix Vista?

Illustration: Gordon Studer
When it comes to Vista, the fix is in. It's just not the fix many mainstream users are looking for.

Last fall, Microsoft announced that Vista Service Pack 1 would improve the operating system's performance. The company claimed that SP1 would rev up Vista in the areas of file copying (see "Microsoft Details Plans for Vista SP1--Beta 'In a Few Weeks'"), waking from sleep or hibernation, and adding or extracting files from compressed folders (see "Closer Look: Beta Windows Vista Service Pack 1").

Microsoft also anticipated that SP1 would strengthen system reliability, broaden support for new hardware and emerging standards, and introduce improvements for IT administrators. We didn't attempt to gauge the validity of those claims following Microsoft's mid-March release of Vista SP1 (see "Windows Vista SP1 Available Now--Do You Need It?"). Instead we tested systems configured with and without Vista SP1 to see whether the promised performance gains would materialize.

What We Tested and Why

Note: Tests conducted by the PC World Test Center. Results in bold red are the best for that test and system.
Note: Tests conducted by the PC World Test Center. Results in bold red are the best for that test and system.
We measured SP1's speed using our 32-bit WorldBench 6 test suite. We compared the overall performance of Vista Business with SP1 to that of Vista Business without SP1 and to that of Windows XP Professional with SP2.

Our test results indicate that where SP1 outperformed Vista, the margin of improvement was slim. What's more, we found little evidence that SP1 handled file copying better; amazingly, in many cases, pre-SP1 Vista was speedier.

Vista SP1 didn't wake from hibernation on our test desktop much faster than pre-SP1 Vista did, either--and on our laptop, it was slower to wake. SP1 did beat Vista at compressing files, but by a tiny margin. Its only clear win over old Vista was in decompressing files--and even there, it was slower than XP.

Many readers (see "PCWorld.com Users Say: Save Windows XP!") have taken Vista to task for using too many system resources on its Aero environment, for trying to appeal to both business users and consumers without fully serving either group, for problems with unavailable or limited drivers, and for general bugginess.

Unfortunately, we couldn't evaluate SP1's impact on each of those shortcomings individually, so we used our WorldBench 6 benchmark tests to gauge how well XP, Vista, and Vista SP1 performed a standard set of tasks. We found that Windows XP still outperforms Vista--with SP1 or without--by a wide margin. SP1 finished second in many cases and third in others.

Given those results--and in view of the bug fixes SP1 delivers--current Vista users should take the free service-pack upgrade. But if you still use XP, you may want to stick with it for now.

How We Put Windows to the Test

Note: This is a sampling of all hand-timed test results. Lower times are better.
Note: This is a sampling of all hand-timed test results. Lower times are better.
We conducted three types of performance evaluations: automated WorldBench 6 tests, hand-timed tests for file management and waking from hibernation, and video frame-rate tests using popular games. We ran WorldBench 6 with Windows XP Professional SP2, Windows Vista Business, and Vista Business SP1 on a higher-end desktop and a low-end but Vista-ready laptop. (For each system's specs, see "WorldBench 6 Results: XP vs. Vista vs. Vista SP1.") We ran each test three times, and then averaged the scores. To learn how our WorldBench 6 tests work, consult "WorldBench 6 Beta 2 FAQ, Version 1.1."

To test video, we measured the frame rate (in frames per second) that each system could handle from two games, Doom 3 and Far Cry. We ran tests with antialiasing on and off, and at the 1024 by 768 and 1280 by 1024 resolutions.

Finally, we ran hand-timed tests to measure file copying, file compressing and decompressing, and recovery time from hibernation on both systems.

XP Wins Convincingly

As the composite scores indicate, XP outperformed both Vista and Vista SP1 on our automated WorldBench 6 tests by a significant margin, on the desktop and again on the notebook. Vista SP1 finished slightly ahead of plain Vista on the desktop test bed, but managed only a tie with it on the laptop.

In the areas where Service Pack 1 improved on pre-SP1 Vista's performance, the gains were far from dazzling.

In our desktop WorldBench 6 tests, XP, which dominated most of the re sults, finished last on a couple of measures: the Nero CD-burning test and the Photoshop image-effects test. The latter marked the only time Vista SP1 emerged as the clear winner.

In the Nero test for desktops, Vista ran 4 percent faster than Vista SP1 and 12 percent faster than XP.

On the laptop, XP had the best score on every test. The laptop results indicate that, as we expected, XP is the better choice for lower-power PCs.

On both test beds we used, the SP1 results were frequently the same or extremely close to the results for Vista. But the Photoshop and Firefox tests showed little consistency in scores between the desktop and laptop PCs running the two Vistas. Hardware differences definitely influenced the outcome.

Windows, Do You Copy?

The file-copying tests yielded a hodge-podge of results that didn't radically favor any particular OS. According to Microsoft, Service Pack 1 improves the operating system's file-copying times; but in several of our tests, plain old Vista beat Vista with SP1. Vista without the update was about 10 percent faster than Vista with SP1 at copying a .zip file over a network, for example.

Results differed when the files being copied were scattered across the hard drive; in those tests, which measure both throughput and access time, SP1 outperformed original Vista by a few seconds on the desktop. XP finished last, trailing the winner by 15 seconds. On the laptop, however, pre-SP1 Vista came in first, by several seconds.

Access time played a lesser role when we copied a 3.06GB .zip file on a local drive. On the desktop computer, Vista was about 13 percent faster than XP and about 16 percent faster than SP1. On the laptop, Vista was 18 percent faster than XP and 19 percent faster than SP1. At copying .zip files, original Vista outshines both XP and SP1.

SP1 took top honors in only one heat of our hand-timed evaluations: the multicopy test on the desktop--and that by a negligible margin over plain Vista.

In compression tests, SP1 did improve on Vista's performance, but XP stole the show. Vista took three times as long as XP to extract files on the desktop, and SP1 took twice as long.

SP1 also outran Vista when adding files to a compressed folder, but only by about 2 percent. XP narrowly won the competition on both test systems.

In our hibernation tests, SP1 woke up faster than Vista by less than a second on the desktop and by 3 seconds on the laptop. As in the compression tests, XP took first place on both test systems, but its margin of victory over SP1 was narrower on the desktop system (2 seconds) than on the laptop (9 seconds).

The Frame Game

Note: Desktop tests at 1280 by 1024 resolution. Higher numbers are better.
Note: Desktop tests at 1280 by 1024 resolution. Higher numbers are better.
Gamers have complained that Vista hampers game play, and our tests confirm that XP is substantially faster than either Vista version on games. SP1 had almost no gain over original Vista.

In frame-rate tests on the desktop PC, XP bested both versions of Vista every time. On the Doom 3 tests, antialiasing usually had a slightly negative effect on Vista's performance, but antialiasing scarcely made a difference to XP. In the desktop Doom 3 tests without antialiasing, XP's frame rate was about 14 percent faster than that of second-place SP1; with antialiasing turned on, XP's frame rate was about 14 percent faster than that of second-place Vista. In the Far Cry tests, Vista and Vista with SP1 improved a bit with antialiasing turned on. Antialiasing degraded XP's performance on the Far Cry test, though XP still won handily on every test.

Our test laptop wasn't built for gaming. XP, Vista, and SP1 fared about the same; SP1 and Vista bested XP in Doom 3 and matched XP in Far Cry. But the frame-rate counts were very low for all three, ranging from 2 to 7 fps with antialiasing and 10 to 25 fps without.

As these tests show, graphics performance depends greatly on hardware and on the OS. So how do you know whether your current (or next) PC's graphics card is up to snuff?

The Rock-Bottom Line

It's not unusual for an OS designed for older, slower PCs to test well; newer operating systems tend to have more-sophisticated features designed for more-powerful machines. For example, Vista's Aero environment requires a lot more graphics power than XP's interface does. But our tests also indicate that Vista SP1 doesn't improve markedly over the original Vista--even when both are installed on a high-powered PC with components built to run the new OS efficiently. SP1 did outperform plain Vista in a few areas, but not in any jaw-dropping way. What's more, the service pack's poor performance in many areas compared with Windows XP shows that Vista is still playing catchup.

Unfortunately, you may not be able to avoid Vista forever. Windows XP is slated to disappear from shelves in June, along with the last new computers that have XP installed at the factory.

In addition, as computer makers direct more resources toward Vista, cus tomers will be harder-pressed to find XP-compatible drivers for new hardware components. So if you want an XP system that will last as long as possible, you need to start your shopping today while such PCs are still available.

What We Really Want From Our Relationship With Vista

Illustration: Gordon Studer
Dearest Microsoft,
We need to talk. We, the people of the PC World forums and the Vista user base, waited patiently for Vista to finally come out. We believed you when you said Service Pack 1 would fix most of Vista's shortcomings. Then we waited patiently again for SP1 to arrive.

And now, well, we're sick and tired of waiting--not just for Vista to get better, but for our Vista PC to finish booting up. And after waiting that out, we're fed up with having to "allow" various things to happen by clicking pop-up screens.

The thing about pop-ups is that we pay attention to them only when they're unexpected. In Vista, they cry wolf so abundantly that we start clicking 'Allow' by default, which leaves us no more secure than we'd be without pop-ups. That's why many of us disable the pop-ups in our User Account Control settings.

But it's not all about security. We especially don't like pop-ups that obstruct us from rearranging our Start menu and ask us if we're sure we want to do anything. Trust us, we're sure.

It's almost like you don't trust us. Now, we don't blame you for being a little paranoid because of past Windows attacks. But what about the good guys who just want to get stuff done?

Just ask "Evildave" of the PC World forums. He's frustrated because he can't tweak Vista's Registry, even when he's logged in as an administrator. In fact, Evildave got so frustrated by Vista that he dumped Windows--not for a Mac, but for Ubuntu Linux.

Meanwhile, "Lilxkid24" is irked by the lack of drivers for his gear, by the drop in frame rates between XP and Vista, by the constant RAM-hogging even when his PC is idle, and by the lack of compatibility with older software and games. He has a laundry list of gripes on the PC World forums. You should take a look.

Don't get us wrong. We to love Vista. In fact, we love parts of Vista. Senior Editor Yardena Arar, for example, loves the pop-up previews of pages in the taskbar and the seamless way Vista syncs with her Windows Mobile devices. Senior Writer Darren Gladstone loves Vista's integration with his Xbox 360. Working as a Media Center Extender, it tracks sports stats while he watches a game--though he longs for better native video codec support over his Xbox 360 for viewing AVI files.

Vista just isn't what we want it to be after buying a system with enough RAM, battery life, and processing power to run it.

Basically, we want XP back. We liked XP. Can you hook us up?

Love,

A large portion of the Windows user base

This "open letter" is based on forum comments posted in reponse to our request for Vista gripes (see "Your Feedback: An Open Letter to Microsoft About Vista"). For more user feeback on Vista, see "PCWorld.com Users Say: Save Windows XP!"

Tim Moynihan

Will Windows 7 Be the Vista You've Always Wanted?

Illustration: Gordon Studer
Microsoft is keeping details of the next version of Windows close to its vest. But recent company moves and revelations hint at what we may see in the next operating system, code-named Windows 7 and due for release in late 2009 or early 2010.

At its recent MIX 08 conference in Las Vegas, Microsoft unveiled Internet Explorer 8, technology that likely will be a part of Windows 7, though Microsoft hasn't linked the two products yet.

Microsoft demonstrated IE 8, showing mainly developer improvements, but also some new end-user features. One, "Activities," lets users highlight any word or phrase on a site and then choose from a drop-down list of actions to take using that term, such as doing a Live Search or searching MSNBC for more information.

Aside from IE 8, little about Windows 7 is on the record, and the word from Microsoft is that the company will talk about the OS when it's good and ready. But analysts warn against expecting Windows 7 to be a blockbuster release, given the fallout Microsoft endured for its late and, to many, disappointing Windows Vista.

Mike Cherry, an analyst with Directions on Microsoft, says that for Windows 7, Microsoft's developers will likely keep the fripperies to a minimum. Doing so will allow them to deliver "something reasonable they can complete by a reasonable date," he says. "Their goal will be to try to put Vista behind them."

Recent court papers in a class action suit over Microsoft's Vista Capable sticker program revealed that even Microsoft executives were having driver- and application-compatibility problems with Vista after its release. In those papers, executives made statements to the effect that the delays and problems that accompanied Vista's rollout won't happen again.

Microsoft is still struggling to persuade businesses to move from XP to Vista, in light of its compatibility problems and hardware requirements. Some business users have suggested that companies may skip Vista altogether and hold on to Windows XP for a little longer before migrating directly to Windows 7.

To remedy these problems, Cherry says, Microsoft should focus on making Windows 7 a stable release for business customers, based on the same code used in the recently released Windows Server 2008. He believes that Microsoft's mistake with Vista was to try to serve both private consumers and business customers with a flashy release that added a host of multimedia functionality at the expense of practical considerations such as performance and compatibility. "What we really need [is] for a business edition to be built off of that server code, so it would look much less fancy than Vista--much more austere with not a lot of wasted functionality," Cherry says.

Elizabeth Montalbano

Subscribe to the Windows Tips & Trends Newsletter

Comments