Chinese vendors Tencent and Qihoo have been accused of gaming antimalware tests by releasing optimized versions of their products for testing purposes that wouldn’t be reflected in the real world, several antimalware firms said.
Both companies will be stripped of their rankings, AV‐Comparatives, AV-TEST and Virus Bulletin said. The three stripped the rankings of Qihoo last week, and of Tencent on Monday night.
Both Tencent and Qihoo finished among the top six antimalware vendors in AV-Test results released in March, which ranked the firms on protection, the performance of the engines, and how usable they were. But they will apparently no longer appear on the official list, just as a college basketball team can be retroactively stripped of their victories and individual awards if improprieties are involved.
Why this matters: No one likes a cheater, but cheating on antivirus benchmarks potentially puts users at risk, if the product they choose doesn’t actually provide the protection it seems to. The testing firms are also protecting their own credibility in rating these products.
‘Inappropriate behavior’
“Today, three of the world’s most renowned and trusted security testing bodies, AV‐Comparatives, AV-TEST and Virus Bulletin, stand united to censure a security vendor after finding the firm submitted products for comparative and certification testing which behaved significantly differently from those made available to its users and customers,” the three firms said in a joint statement posted to AV-Comparatives’ Facebook page, in censuring Qihoo.
The three companies published a joint statement on the “inappropriate behavior” of Qihoo 360. In the tests, the antimalware organizations claimed, all products submitted for testing by Qihoo had one of the product’s four available antimalware engines, provided by Bitdefender, enabled by default, while a second, Qihoo’s own QVM engine, was never enabled. That differed in the product that Qihoo released to the public. In that, the firms claimed, the Bitdefender engine was disabled and the QVM engine active.
“According to all test data this would provide a considerably lower level of protection and a higher likelihood of false positives,” the firms claimed.
Qihoo, for its part, reportedly said the tests were outdated, too one-dimensional and do not reflect the different online conditions and behaviors in different countries and regions.
At the time, Qihoo accused Baidu and Tencent of similar practices. The three antimalware firms said they found “some unexpected flags within [Baidu and Tencent] products, marked with the names of several test labs and implying some difference in product behavior depending on the environment they were run in.”
However, only Tencent has been found to be gaming the antimalware tests, according to the antimalware firms.
Virus Bulletin noted that “these optimizations, which have been found in all recent public versions of the products, provide minimal benefit to normal users and could even degrade the level of protection offered by the products,” the company posted to its Facebook page, specifically referring to Tencent.
All three labs will be “imposing stricter controls on participants to reduce opportunities for such actions” in the future, they said.