Chinese anti-virus vendor Qihoo 360 has been stripped of all of the certifications awarded to it this year by the three leading anti-malware testing agencies, after being found to have broken the rules.
In a joint statement issued by AV-Comparatives, AV-Test.org and Virus Bulletin, Qihoo was found guilty of attempting to game detection tests:
Investigations by the three labs found that all products submitted for testing by Qihoo had one of the product's four available engines, provided by Bitdefender, enabled by default, while a second, Qihoo's own QVM engine, was never enabled. This included versions posted to ostensibly public sections of the company’s websites.
By contrast, as far as can be determined, all versions made generally available to users in Qihoo's main market regions had the Bitdefender engine disabled and the QVM engine active. According to all test data this would provide a considerably lower level of protection and a higher likelihood of false positives. Options are provided in the product to adjust these settings, but as the majority of users leave settings unchanged, most tests insist on using the default product settings to best represent real‐world usage.
On requesting an explanation from Qihoo 360 for their actions, the firm confirmed that some settings had been adjusted for testing, including enabling detection of types of files such as keygens and cracked software, and directing cloud lookups to servers located closer to the test labs. After several requests for specific information on the use of third‐party engines, it was eventually confirmed that the engine configuration submitted for testing differed from that available by default to users.
In short, Bitdefender's anti-malware engine was much better at reliably detecting the malware in tests done by the testing agencies, but - in Qihoo's home market of China - Bitdefender's engine was disabled by default, and Qihoo's own is used instead.
Here is what John Hawes, chief of operations at Virus Bulletin had to say:
"This sort of thing doesn’t really help anyone. Independent tests serve both users and developers, showing which products are performing best and highlighting areas where developers need to work harder. If the products being tested aren't those being used in the real world, nobody's getting any useful information."
Qihoo predictably wasn't best pleased with the accusation that it had been caught cheating, and counter-claimed that its Chinese rivals Baidu and Tencent had also submitted versions of their software to testers that were different from those offered to the public.
However, although the testing agencies confirmed a difference in Baidu and Tencent's products, it also determined that the products gained no advantage from it. Furthermore, both Baidu and Tencent are said to have provided acceptable reasons for the differences.
Qihoo is now suffering the consequences of attempting to manipulate the test results.
Interestingly, this may not be the end of the story.
An update posted on Facebook by AV-Test.org reveals that another anti-virus (not Qihoo) may have been cheating in a different way - manipulating its performance in speed tests by excluding scanning of certain file types:
Unfortunately we have to post an update to our current findings. So far we checked the possible manipulation of our protection tests. This is what is being reflected in the posted statement.
We have now started to evaluate the possible manipulation of our performance testing. We have found strong evidence that another company, not Qihoo, is optimizing their product to do well in our performance test by excluding certain files and processes from checking. This is based on filenames and process names and can pose a security risk as well! We will check with AV-Comparatives and VB100 to verify our findings and will let you know as soon as we have the final data.
Get the popcorn folks...