Revealed: The anti-virus vendor cheating in independent tests

Chinese anti-virus vendor Qihoo 360 has been stripped of all of the certifications awarded to it this year by the three leading anti-malware testing agencies, after being found to have broken the rules.

Qihoo 360 Total Security

In a joint statement issued by AV-Comparatives, AV-Test.org and Virus Bulletin, Qihoo was found guilty of attempting to game detection tests:

Investigations by the three labs found that all products submitted for testing by Qihoo had one of the product's four available engines, provided by Bitdefender, enabled by default, while a second, Qihoo's own QVM engine, was never enabled. This included versions posted to ostensibly public sections of the company’s websites.

By contrast, as far as can be determined, all versions made generally available to users in Qihoo's main market regions had the Bitdefender engine disabled and the QVM engine active. According to all test data this would provide a considerably lower level of protection and a higher likelihood of false positives. Options are provided in the product to adjust these settings, but as the majority of users leave settings unchanged, most tests insist on using the default product settings to best represent real‐world usage.

On requesting an explanation from Qihoo 360 for their actions, the firm confirmed that some settings had been adjusted for testing, including enabling detection of types of files such as keygens and cracked software, and directing cloud lookups to servers located closer to the test labs. After several requests for specific information on the use of third‐party engines, it was eventually confirmed that the engine configuration submitted for testing differed from that available by default to users.

QihooIn short, Bitdefender's anti-malware engine was much better at reliably detecting the malware in tests done by the testing agencies, but - in Qihoo's home market of China - Bitdefender's engine was disabled by default, and Qihoo's own is used instead.

Here is what John Hawes, chief of operations at Virus Bulletin had to say:

"This sort of thing doesn’t really help anyone. Independent tests serve both users and developers, showing which products are performing best and highlighting areas where developers need to work harder. If the products being tested aren't those being used in the real world, nobody's getting any useful information."

ChinaQihoo predictably wasn't best pleased with the accusation that it had been caught cheating, and counter-claimed that its Chinese rivals Baidu and Tencent had also submitted versions of their software to testers that were different from those offered to the public.

However, although the testing agencies confirmed a difference in Baidu and Tencent's products, it also determined that the products gained no advantage from it. Furthermore, both Baidu and Tencent are said to have provided acceptable reasons for the differences.

Qihoo is now suffering the consequences of attempting to manipulate the test results.

Interestingly, this may not be the end of the story.

An update posted on Facebook by AV-Test.org reveals that another anti-virus (not Qihoo) may have been cheating in a different way - manipulating its performance in speed tests by excluding scanning of certain file types:

Statement from AV-Test.org

Unfortunately we have to post an update to our current findings. So far we checked the possible manipulation of our protection tests. This is what is being reflected in the posted statement.

We have now started to evaluate the possible manipulation of our performance testing. We have found strong evidence that another company, not Qihoo, is optimizing their product to do well in our performance test by excluding certain files and processes from checking. This is based on filenames and process names and can pose a security risk as well! We will check with AV-Comparatives and VB100 to verify our findings and will let you know as soon as we have the final data.

Get the popcorn folks...

Tags: , , , , ,

Smashing Security podcast
Check out "Smashing Security", the new weekly audio podcast, with Graham Cluley, Carole Theriault, and special guests from the world of information security.

"Three people having fun in an industry often focused on bad news" • "It's brilliant!" • "The Top Gear of computer security"

Latest episode:

, , , , ,

12 Responses

  1. Ambi

    May 1, 2015 at 5:22 pm #

    You've gotta love Qihoo's response, pointing the finger at their rivals, "Well…they're doing it too…", like spoiled brats who got caught with their hands in the cookie jar.

    But seriously, is anyone actually surprised? They're communists, after all. They live in a culture that reviles integrity.

  2. david L

    May 1, 2015 at 8:13 pm #

    Popcorn indeed! I have been warning people about various Chinese products to no avail, the list of things by numerous software/hardware developer's is a long one. But most just don't care about their privacy,or the theft of personal information evidently. Take for instance any apps that are in playstore by Chinese developers are most likely connecting to server's in China over http. Mostly Baidu analytics. But because of Great Cannon,these connections are extremely vulnerable. Not only that,but https is also vulnerable because China has a certificate authority and could decrypt those https connections. They were recently caught abusing that authority,and as a result,Google and Mozilla are removing those CNNIC certificates from all their products. And all you Chinese phone lover's,guess what? Your phones are just as vulnerable as all those apps. Thhey too call home. Do your homework folks,don't take my word.

  3. Anonymous

    May 1, 2015 at 9:06 pm #

    What A/V do you use Graham? I've been with ESET now for a number of years (after Norton/McAfee rubbish) and couldn't be happier.

    • Anonymous 1 in reply to Anonymous.

      May 1, 2015 at 9:24 pm #

      'Anonymous', the three leading internet security packages are: Kaspersky, Bitdefender and Norton.

      Personally I would recommend Kaspersky Internet Security. It always comes out at the top of every independent test, is rated 'Platinum' in terms of its protection, has the fewest false positives (0) and the highest detection score (100). It protects from webcam hijacking, keyloggers, email spam, webpage advertising and you can choose to run only whitelisted applications if you're paranoid. It's got a crazy amount of other features and I for one would highly recommend it. UK customers can get it for free via Barclays if you use their internet banking.

      If you're concerned with your current AV why don't you run this dummy battery of tests through it:

      http://www.amtso.org/check-desktop

      • Danny C in reply to Anonymous 1.

        May 5, 2015 at 1:59 pm #

        Kaspersky is a solid product, no doubt about it. However, not quite as awesome as described. Virus Bulletin (as one of the testers in the article) found it to detect 75% of their samples in their last Windows 7 test. The winner was AVG, with 91%. However, these tests are fairly brutal, including testing for viruses that have come out AFTER the product suite was last updated, for instance.
        It was a similar result in their slightly older Windows 8 test.
        But, this is a bit like arguing about which is the best car in the world. Some want speed, some want comfort, some want the cup holders… Any of the big vendors will give you strong protection.

  4. WhistleBlower

    May 2, 2015 at 5:16 pm #

    This revelation is not about an AV vendor cheating on test results. They all submit "tweaked" engines optimized for the tests and are not versions released with the product. Good scores = more sales. This revelation is about an OEM (Bitdefender) framing a partner because they are not getting paid for their engine – as Qihoo ships the product without it enabled by default. Bitdefender probably only get paid if the end user enables it – but Qihoo use it for test scores = more revenue. Obviously, someone did a bad job of writing the OEM contract and now BD are shaming Qihoo cause they're losing money.

    People that live in glass houses shouldn't throw stones. Most small-time AV vendors would stab their Grandmother in the back for $5. Integrity is a dirty word in the AV industry.

  5. Scott Hartley

    May 3, 2015 at 9:45 pm #

    How the hell is that cheating, its the same exact product if they were to ship it with all engines enabled by default then they would be top of the line all the time.

    This honestly sounds like someone got upset that their product was scoring higher than theirs.

    • Coyote in reply to Scott Hartley.

      May 5, 2015 at 1:59 am #

      "… its the same exact product IF they were to ship it with all engines enabled"

      IF. They didn't, however. 'If' is a very interesting word and funnily enough your usage of it actually answers your question.

      "This honestly sounds like someone got upset that their product was scoring higher than theirs."
      Or you didn't read it carefully enough (or perhaps you misread or misinterpreted parts of it). Quoting parts of the statement and made relevant words UPPERCASE:

      "Investigations by the three labs found that ALL PRODUCTS SUBMITTED FOR TESTING by Qihoo had ONE of the product's four available engines, PROVIDED BY BITDEFENDER, enabled by default, WHILE a SECOND, QIHOO'S OWN QVM engine, was NEVER enabled. This INCLUDED versions POSTED TO ostensibly PUBLIC SECTIONS of the company’s websites.

      By CONTRAST, AS FAR AS CAN BE DETERMINED, ALL VERSIONS made GENERALLY AVAILABLE to users in QUIHOO'S MAIN MARKET REGIONS had the BITDEFENDER engine DISABLED AND the QVM ENGINE ACTIVE."

      So yes, it clearly is manipulation and therefore cheating. If it was their own engine then why didn't they have it enabled to the public ? Well there's the answer – their engine IS enabled for the public, EXCEPT that the engine enabled for TESTING is DIFFERENT. The fact remains the engine tested is not the same thing as is for general use, and therefore the test isn't legit.

      (Also, since they responded by accusing other vendors instead of addressing the problem directly, it isn't denying it, either. That is incriminating)

      • Coyote in reply to Coyote.

        May 5, 2015 at 10:17 pm #

        Oh, and just to point out. I'm not judging their testing methods. I'm pointing out what they're saying, why they see it as cheating. Whether they're cheating or not is definitely – as ever – up to the definition of cheating. But those making the rules (those running the tests) are those who define cheating. Still, I do find it troublesome that they would have one mode enabled as testing but not have it in live systems. If they're all for the other engine why not have it in the live systems ? More to the point, I find it odd they would have it different. If they're perfectly fine with it in testing, and it does better than what they have in live, why not use the test engine in the live engine too ?

    • WhiteTiger in reply to Scott Hartley.

      September 5, 2015 at 12:58 pm #

      well.. actually I've been using the Bitdefender engine myself and everything turned off. Yet you don't seem to be able to turn the QVM engine off. I get a lot of "heuristic" false-positives this way and I'm basically ignoring every popup that tells me "HEUR" as the type of virus.. (basically having to manually restore the detected file from the quarantine)

      So no, the end product doesn't even to be tweak-able to the same level. Besides most users use the default anyway. (thinking they are protected as AV-Test and others "proved")

      Though AV-Test is actually to blame… if they really ask the AV vendor for a version to use, they're already dump themselves… A test does of course require the version from their page that every customer uses.
      It might be ok to ask which version to use… eg. the latest one got a minor issue and it's best to use the previous one… but you should never test the software that the vendor "prepared" for you. (and it's quite likely that most cheat here, that's what every bigger company does behind the scenes)

  6. Sandro

    September 2, 2015 at 6:00 pm #

    Well, talking about antiviruses then I believe the most important is the detection precision and lowest score of false positives… Antivirus vendor could ONLY be cheating with those previously named results! NB! Qihoo has ALWAYS had their Qihoo engine installed by default + they have four different modes of security: Performance, Balanced, Security and Custom. So in Performance and Balanced mode only Qihoo engine is doing a job + additional services. Additionally to default Qihoo engine Bitdefender and Avira engines are being automatically installed, when a customer have chosen Security mode! Its really a customer´s choice! So Qihoo offered them for testing a Security mode engine, where all three engines Qihoo, Bitdefender and Avira installed!!! And you call it a cheating!? For me the its rather the testers are cheaters because they deliberately ruining Qihoo´s reputation with false accusations protecting other famous vendors interests in competition, who are obviously losing this competition to Qihoo!!! Another contradictionary information I found in the article is that Qihoo has in total 4 antivirus engines, when it always has been 3 engines: Qihoo, Avira and Bitdefender (by the way no word was said about Avira engine in the article… Thats strange!!!)… I am just surprised of those testers unprofessionalism saying it has 4 engines + on the top of everything they accuse Qihoo in cheating!!! Really dirty game is going on!

    Next, I have been a user of 360 Total Security for over 1 year and I can tell you from my experience that it is really the best product on the market… Previously I have been using Fsecure, Bitdefender, Avast and Eset and they all allowed the backdoors for viruses… The situation got so ridiculous that my windows updates were not installing anymore until I understood that I am trusting the very wrong vendors! I have made a huge research in the internet, found out that from alternative testers that those were giving the best result to Qihoo, exchanged my "famous" vendor to Qihoo and here we are – my computer has been staying clean and strong agains all kind of viruses for over the year, and my windows updates started to work!!! The thing is that when you install Qihoo it also scans all your PC for security and other vulnerabilities, then when those detected patches your PC with all missing security updates from Microsoft + offers also optional ones… Anyway, this is how I got rid of viruses thanks to Qihoo and my PC working fine! Additionally to forementioned Qihoo has a perfect system optimizing and cleaning tools by default for your PC! Qihoo is just really the best product on the market and it washes all others out from the waters! Just believe me! The current problem what is going on around Qihoo is just a dirty war of its competitors!

  7. Sandro

    September 2, 2015 at 6:10 pm #

    Forgot to mention that why those testers are not accusing for instance all their supported "famous" vendors in cheating because all of them have also different products, such as for instance Free, Internet and Global versions, which all have different levels of protection!? I mean, its obvious, that Free product will not do the same jom as Global version!!! Why in this case only Qihoo being discriminated!? Savages – i dont have any other words to describe the attitude of those autors of the article and testers! Its great that Qihoo have all those 3 modes available within the one application, which their competitors have as different products! Just shameful attitude to the best ever vendor, whoi is offering its great product to the public FREE of charge!

Leave a Reply