Extended Beta May Stabilize Office 2003

Microsoft's decision to delay Office 2003 to squash more bugs could produce a more solid release, and may void the old joke about savvy consumers skipping the company's version 1.0 products.

Microsoft on Thursday said it will "refresh" the current beta test version of Office 2003 before freezing the code, which will delay the suite's release until the third quarter.

The upshot? The initial Office 2003 might be as stable as most Microsoft products after the first Service Pack update, says Rob Enderle, a research fellow with Forrester.

"I can speak for myself and most of my clients: We'd rather have something come late and work than come on time and be a pain because it isn't really finished," Enderle says.

Errors Captured

Both the analyst--who is among the estimated 600,000 beta testers--and Microsoft attribute the timetable change to the enormous feedback enabled by Microsoft's automated error-reporting system in the Office 2003 beta edition. The utility pops up whenever the software falters, inviting the user to send an automated report online to Microsoft.

The error report merely sends the failed code, and does not identify the user, says Eric LeVine, program manager for Microsoft Office. Automated error reporting is also part of the shipping versions of Windows XP and Office XP. Microsoft estimates that about one-third to one-half of the time, users agree to send the feedback.

This is the first time it has been part of an Office suite beta. And because the users are testing a prerelease product, they're more likely to click "send" to an error-report request, he says.

"We've been trying to make it very, very easy for customers to take one of the most frustrating things that can happen with a computer, when an application fails, to get real-world feedback to us so we can address the problem," LeVine says.

Improved Reporting

Microsoft has also improved the error-reporting utility: it now permits users to delay sending the report if they're offline when it occurs.

Enderle says he's not aware of any other developers automating the error-reporting process, but expects competitors will adopt similar systems, given Microsoft's success.

"They undoubtedly captured a whole lot of problems they didn't expect," says analyst Enderle. Traditional beta testing relies on users taking the initiative to report problems. "With the automated method, they're capturing almost everything," he adds.

Office entered beta 2 in March. It is a so-called public beta, available for download or on request by any interested parties. Microsoft expected about 500,000 testers.

"This is the largest beta test for Office," LeVine says. "And the larger we can make a beta, the more we can approximate real-world experience, and the more confidence we have that we're seeing the top issues and that the fixes we make really hit this on the head."

Diverse Bugs

LeVine declines to specify the kind of bugs that are getting the most attention, but says they've been distributed across the multi-application suite. He notes that "the 20 percent of most common defects accounted for 80 percent of the real-world problems" in the initial use of the automated error checking.

"It showed us that in the past there were defects [in] our traditional ways of doing betas [that] weren't catching all the problems." Now, he says, Microsoft hopes to fix more bugs before the product ships. However, he also notes that more than half the errors reported from Windows XP involved third-party applications, not the operating system.

With Office 2003's beta test, that includes Microsoft's own applications. "For InfoPath and OneNote, this is an amazing opportunity," LeVine says of the two new members of the Office line. "These products have never shipped before, and so we're getting feedback--we're getting real-world feedback before it ships."

It's the way things should be, says analyst Enderle.

"This is how it's supposed to work--it's better that they delay and make sure the product works before it ships," he says.