NVidia, ATI Accused of Altering Benchmarks

An organization that produces benchmark tests for measuring the performance of graphics chips has updated its test after it found code in Nvidia's graphics chip drivers that detected certain tests and altered the chip's performance to inflate results.

Futuremark, which developed the 3DMark03 benchmark, posted a statement on its Web site Friday detailing allegations that some Nvidia drivers have "detection mechanisms" that trigger a higher level of performance when certain tests within 3DMark03 are detected. ATI Technologies, Nvidia's rival, also appears to have altered its drivers in order to boost test results, Futuremark said.

Futuremark said it identified eight instances where Nvidia's Detonator FX 44.03 and 43.51 WHQL drivers detected specific 3DMark03 tests and inflated the benchmark results to the detriment of overall image quality. The allegations were first lodged by ExtremeTech, a hardware enthusiast Web site.

Tough Competition

3DMark03 has been criticized by some as favoring products from ATI over those of Nvidia, said Peter Glaskowsky, editor in chief of the Microprocessor Report in San Jose, California. The benchmark contains certain DirectX calls that favor ATI, he said, and Nvidia appears to have been trying to rectify the situation by altering its drivers to convert that type of DirectX call into one that works better on Nvidia hardware, Glaskowsky said.

DirectX is an API (application program interface) developed by Microsoft to help games run on Windows operating systems, and 3DMark03 is designed to measure performance on hardware running DirectX 9.0, the latest version.

"It looks like what Nvidia is doing is trying to counteract their disadvantage, but some of the methods described [by Futuremark] aren't the correct methods for dealing with that problem," Glaskowsky said.

In some instances the 3DMark03 "shaders"--pieces of code that render the appearance of a surface, such as a road or a tree--are discarded in favor of ones found in the Nvidia drivers that work more efficiently, according to Futuremark's statement. In other cases, code in the drivers artificially reduces the workload demanded by the benchmark test, thereby increasing performance, Futuremark said.

Nvidia's Side of the Story

Nvidia, based in Santa Clara, California, did not respond to specific questions about Futuremark's allegations, choosing instead to issue a statement.

"Since Nvidia is not part of the Futuremark beta program--a program which costs hundreds of thousands of dollars to participate in--we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer," the statement said. "We don't know what they did but it looks like they have intentionally tried to create a scenario that makes our products look bad."

A new version of 3DMark03 with slightly different code that blocks the Nvidia drivers from detecting specific tests reduced the benchmark score of a system with Nvidia's GeForceFX 5900 Ultra and the 44.03 driver by 24 percent, Futuremark said.

Making Alterations?

In fact, ATI appears to have also been altering its software drivers to improve its test scores, Futuremark said. The new benchmark reduced the performance of a system with ATI's Catalyst 3.4 driver and the Radeon 9800 Pro by almost 2 percent. This was due to a drop in performance of some 8 percent for one particular part of the test, suggesting that ATI's drivers also had been making adjustments to produce better results. 3DMark03's margin of error was 3 percent.

ATI's drivers make use of an "optimization" that reorders the 3DMark03 instructions for DirectX 9.0 to take advantage of its architecture, said Patricia Mikula, public relations manager for ATI. Changing the code order isn't cheating, because ATI's drivers still produce the same result, just in a different way, according to Mikula.

Nevertheless, ATI will remove the optimization from an upcoming driver release to avoid creating the perception that it is doing anything wrong, Mikula said.

Playing Games

Chip makers love to cite benchmark results in marketing materials when announcing a new chip. Benchmarks are virtually the only method of comparing performance of competing chips or systems, but are often criticized for not mirroring real-world performance.

For that reason, gaming benchmarks carry more weight among certain groups of PC buyers. Benchmarks exist for popular PC games such as Doom, Quake, and others. If a user is interested in a specific game or types of games, he or she can purchase a graphics card or system that wins that performance crown for that specific game.

But according to Futuremark, benchmarks devised to measure the performance for particular games don't demonstrate the overall performance of the hardware, and are more ripe for exploitation by driver cheats than Futuremark's own tests.

"What you want in a benchmark is something that correctly reflects the performance of games, not just the ones now, but the ones that haven't been developed yet," Glaskowsky said. 3DMark03 was supposed to do just that, but it doesn't accurately portray the performance of both companies' hardware, he said.