How do you test an antivirus utility to make sure it works? Well, one simple method is to present it with thousands of malware samples and see which ones its scan detects. Yes, this kind of static detection is just one layer of antivirus protection, but it can be the first line of defense.
Of course, you also need to make sure the antivirus doesn't erroneously declare valid programs to be malware. Otherwise an antivirus utility that marked every single program as malicious could earn an unwarranted perfect score. The file detection test performed by researchers at AV-Comparatives takes into account both detection accuracy and avoidance of false positives. A dearth of false positives this time around raised scores for quite a few vendors.
The researchers presented each of 21 antivirus products with over 160,000 malware samples, and credited detection either by on-access scan or on-demand scan. AV-Comparatives doesn't disclose the precise number of files used in the false positives test, "because some people tend to calculate a percentage out of it, which is nonsense and misleading." However, in past instances of this test the number of false positives for some products has come close to 100, so the total number of valid files must be quite a bit more than that.
Much Improvement
AV-Comparatives assigns a rating of Standard to any product that passes the test; those that don't pass are rated as just Tested. The very best products can earn an Advanced rating, or even Advanced+. However, a product that exhibits many false positives can drop by one, two, or even three ratings. A product that hits the "crazy many" level (more than 100 false positives) can't pass the test, no matter how good its virus detection.
In the previous round of testing, five products showed enough antivirus accuracy to rate Advanced+, but dropped down to Advanced due to false positives. They are: Avira, BullGuard, Emsisoft Anti-Malware£31.16 at Emsisoft UK, eScan, and Ad-Aware Free. All five managed to hang on to that Advanced+ rating this time around; good going!
Bitdefender Antivirus Plus 2016£19.99 at Bitdefender UK and Kaspersky Anti-Virus (2016)£19.99 at Kaspersky UK routinely achieve top scores with all the labs. Neither had any trouble reaching Advanced+ in this test or the previous one.
Success wasn't universal, though. Poor Baidu Antivirus did well enough in the detection portion of the test to merit an Advanced+ rating. However, "crazy many" false positives knocked it out of the running, down to a failure rating of Tested.
McAfee AntiVirus£34.99 at McAfee UK didn't get into false positive trouble in either test. It was a drop in antivirus detection accuracy that brought it down from Advanced+ to Advanced.
One of Many Tests
In the real world, your antivirus gets numerous opportunities to detect and balk a malware attack. It might block the browser from even connecting to a malware-hosting URLs. It could eliminate the file before it's even downloaded. Or it might identify malware after launch, based on its nasty behavior.
AV-Comparatives runs many different tests, taking into account these different opportunities. There's a real-world dynamic test that does its best to let every protection layer have a chance at success. The retrospective detection test freezes each product's malware signatures and tests them against brand new samples that didn't exist before the freeze. There's even a test that starts with samples that all of the tested products are known to detect, and then measures how well they clean up an infected system.
Different products may succeed at different tests. Webroot£22.49 at Webroot UK, for example, doesn't use static detection, so it's not appropriate for a static detection test. Norton's£198.7 at Amazon UK developers reject the very concept of a static test. Even so, I have to admire products like Bitdefender and Kaspersky that manage to earn top marks from all the labs.
Image courtesy of Flickr User Yuri Samoilov.