How We Test Malware Removal

For every antivirus and security suite review, I perform hands-on testing to see how well the product removes viruses and other malware. I note any problems caused by malware resisting the product's installation and report on what steps were needed to solve those problems. And I use a detailed scoring system to rate each product and compare it against other recently tested products.

Malware-Infested Test Systems
I start with a dozen or more virtual machine test systems, each one pre-loaded with three or four malware samples. These samples include viruses, worms, adware, spyware, Trojans, rootkits, and scareware (rogue security software). I use proprietary analysis software to identify the file and Registry traces installed by the threats on each test system, so I can later check how well the antivirus cleaned them up.

In the past I've separately rated each product's ability to remove commercial keyloggers. I now include keylogger samples only if they're also rootkits, Trojans, or some other form of malware.

On each test system I install the product and manually run an update, to ensure that the very latest virus signatures are used. Using the default configuration, I launch a full scan and take note of which threats the product claims it removed. After cleanup is complete, I use another proprietary tool to measure how successful the cleanup operation was.

Scoring
For each malware sample I assign a removal score from zero to ten. If it didn't detect the threat at all, naturally it gets zero points. Detecting a threat but failing to remove all the executable files earns half credit—five points. I don't ding the product if the only executable left behind is an uninstaller. However, if any of those executable files is still running the score sinks to three points.

To earn the full ten points, the antivirus must remove all executable files and also remove 80 percent or more of the non-executable file and Registry traces. Leaving behind 20 to 80 percent of the junk traces earns nine points. A product that cleans up the executables but leaved 80 percent or more of the non-executable junk gets eight points.

The overall malware cleanup score is simply the average of all the scores for the individual threats. Because rootkits are especially difficult to remove I break out a separate score representing the product's success at rootkit removal. I also break out a separate scareware removal score.

A product's overall rating takes the removal score into account, and the malware blocking score as well. Other factors such as installation problems and system crashes caused by incomplete removal affect the rating too. I also pay attention to the results independent lab tests. Still, high scores on my tests go a long way toward getting a good rating.

About Our Expert

Neil J. Rubenking

Principal Writer, Security

My Experience

When the IBM PC was new, I served as the president of the San Francisco PC User Group for three years. That’s how I met PCMag’s editorial team, who brought me on board in 1986. In the years since that fateful meeting, I’ve become PCMag’s expert on security, privacy, and identity protection, putting antivirus tools, security suites, and all kinds of security software through their paces.

Before my current security gig, I supplied PCMag readers with tips and solutions on using popular applications, operating systems, and programming languages in my "User to User" and "Ask Neil" columns, which began in 1990 and ran for almost 20 years. Along the way, I wrote more than 40 utility articles, as well as Delphi Programming for Dummies and six other books covering DOS, Windows, and programming. I also reviewed thousands of products of all kinds, ranging from early Sierra Online adventure games to AOL’s precursor Q-Link.

In the early 2000s, I turned my focus to security and the growing antivirus industry. After years of working with antivirus, I’m known throughout the security industry as an expert on evaluating antivirus tools. I serve as an advisory board member for the Anti-Malware Testing Standards Organization (AMTSO), an international nonprofit group dedicated to coordinating and improving testing of anti-malware solutions.

The Technology I Use

Much of the testing I do, particularly testing with real-world ransomware, is just plain dangerous. To perform such tests safely, I sequester them inside virtual machines managed by VMWare Workstation. For cross-platform testing, I use a MacBook Air, a Google Pixel 4, and a 6th-generation iPad.

I rely on my Delphi coding skills to create and maintain small applications. These include programs to check whether an antivirus correctly handled the malware it detected, launch dangerous URLs and record the security program’s reaction, and analyze the malware that I collect for use in testing. I also wrote a tiny browser and text editor for use in testing security apps that have predefined reactions for known products.

I do my writing and research on a Dell OptiPlex desktop, relying on Microsoft Word (my fingers know all the shortcuts). Many of my articles include charts and analysis; Excel is my go-to for those. When work hours end, though, I escape the bounds of Microsoft and Windows. There’s an iPhone in my pocket, I relax with my oversized iPad, and my Kindle Oasis is always loaded with the best science fiction and fantasy.

Read the latest from Neil J. Rubenking

Read full bio