How We Interpret Antivirus Lab Tests

When reviewing an antivirus or security suite product, I always perform hands-on testing of the product's ability to clean up malware-infested systems and to protect a clean system against attack. I'm just one person, though, so I can't come near the exhaustive evaluations performed by the independent testing labs. To supplement my own tests I look at results from five major labs, all of them members of the Anti-Malware Testing Standards Organization (AMTSO).

Five Independent Labs
West Coast Labs and ICSA Labs will check the ability of a vendor's technology to detect a vast number of malware samples, and will separately evaluate how well it cleans up the infestation. Virus Bulletin regularly tests security products against their list of viruses in the wild. To attain the VB100 certification, a product must detect all the threats without erroneously flagging any good programs. I look at the ten most recent tests. If a vendor's security technology has achieved VB100 certification in all ten, that's an impressive achievement.

AV-Test.org, based in Germany, keeps inventing new and better tests. Their latest set involves certifying products for antivirus protection under Windows XP, Windows Vista, and Windows 7. Each product gets from 0 to 6 points for protection, repair, and usability, with a total of 12 required for certification. A surprising number of products have failed to reach certification in one or more tests. Others have scored as high as 16 of 18 possible points in all three.

Austrian lab AV-Comparatives.org also keeps inventing new tests, but they run two specific types of test several times a year. The on-demand tests check a vendor's ability to detect a large collection of viruses and other malware samples. The retrospective tests force each product to use virus definitions from before the first appearance of the samples, thus testing the product's ability to detect new and unknown malware. They rate each tested product ADVANCED+, ADVANCED, or STANDARD; occasionally a product fails to even meet the criteria for a STANDARD rating.

Interpreting Results
When looking over results from the labs I have to consider the vendor rather than a specific program. Different tests may use different products or versions from the same vendor, so I take each test as an evaluation of the vendor's technology.

ICSA and West Coast Labs report on a vendor's certification only after success is achieved. Having their certifications is definitely good, but not having them typically means the vendor just didn't choose to participate. Likewise some vendors choose not to participate in Virus Bulletin's testing.

Because of the intensive nature of their testing, AV-Test.org and AV-Comparatives.org typically include just 15 to 20 products in a test. Here again, if a product isn't included I can't count its absence against it. On the other hand, I'm impressed with a product that all five labs consider important enough to test.

Keeping these facts in mind I've devised a system for aggregating test results into a rough overall score. This system may well change as the labs invent new tests, which is why it has to be rough. For AV-Comparatives.org I take the average of the on-demand and retrospective scores, counting ADVANCED+ as 3 and STANDARD as 1. I map the average of the three AV-Test.org results onto a range from 0 to 3. For Virus Bulletin I calculate the percentage of VB100 successes in the ten latest tests and also map that to 0, 1, 2, or 3. Then I average the three ratings. If a product doesn't have at least two of these three ratings I consider that there's just not enough information.

As noted, I have more confidence in a product that's tested by many labs. To account for that, I add to the average one tenth of a point for each lab that tested the product. At present, that's how I reach the aggregate rating of POOR, FAIR, GOOD, or EXCELLENT. This system will surely evolve with time, but I believe it does offer an easy way to summarize what the various labs have to say.

About Our Expert

Neil J. Rubenking

Principal Writer, Security

My Experience

When the IBM PC was new, I served as the president of the San Francisco PC User Group for three years. That’s how I met PCMag’s editorial team, who brought me on board in 1986. In the years since that fateful meeting, I’ve become PCMag’s expert on security, privacy, and identity protection, putting antivirus tools, security suites, and all kinds of security software through their paces.

Before my current security gig, I supplied PCMag readers with tips and solutions on using popular applications, operating systems, and programming languages in my "User to User" and "Ask Neil" columns, which began in 1990 and ran for almost 20 years. Along the way, I wrote more than 40 utility articles, as well as Delphi Programming for Dummies and six other books covering DOS, Windows, and programming. I also reviewed thousands of products of all kinds, ranging from early Sierra Online adventure games to AOL’s precursor Q-Link.

In the early 2000s, I turned my focus to security and the growing antivirus industry. After years of working with antivirus, I’m known throughout the security industry as an expert on evaluating antivirus tools. I serve as an advisory board member for the Anti-Malware Testing Standards Organization (AMTSO), an international nonprofit group dedicated to coordinating and improving testing of anti-malware solutions.

The Technology I Use

Much of the testing I do, particularly testing with real-world ransomware, is just plain dangerous. To perform such tests safely, I sequester them inside virtual machines managed by VMWare Workstation. For cross-platform testing, I use a MacBook Air, a Google Pixel 4, and a 6th-generation iPad.

I rely on my Delphi coding skills to create and maintain small applications. These include programs to check whether an antivirus correctly handled the malware it detected, launch dangerous URLs and record the security program’s reaction, and analyze the malware that I collect for use in testing. I also wrote a tiny browser and text editor for use in testing security apps that have predefined reactions for known products.

I do my writing and research on a Dell OptiPlex desktop, relying on Microsoft Word (my fingers know all the shortcuts). Many of my articles include charts and analysis; Excel is my go-to for those. When work hours end, though, I escape the bounds of Microsoft and Windows. There’s an iPhone in my pocket, I relax with my oversized iPad, and my Kindle Oasis is always loaded with the best science fiction and fantasy.

Read the latest from Neil J. Rubenking

Read full bio