How We Test Antiphishing

Hackers don't always have to attack your computer or plant malware on it in order to steal your personal information. Sometimes they can simply trick you into giving away sensitive data like usernames and passwords at fraudulent Web sites that appear to be PayPal, eBay, your bank, and so on. Security products try to protect against phishing by blocking known fraudulent sites and running heuristic analyses to identify new ones. The real-time analysis feature is very important, because these sites often appear and vanish within just a few days.

To test a product's antiphishing capabilities, I collect a group of suspected phishing URLs from sites devoted to tracking such things. I specifically look for URLs that have been reported but not yet verified as fraudulent. I set up one system protected by the product being tested; one protected by Internet Explorer's built-in antiphishing; and one protected by Norton Internet Security, a consistent antiphishing champion.

Using a small utility I wrote myself, I launch a suspect URL simultaneously under all of the test systems and note the results. I discard any URL that any of the test systems is unable to load. I also discard URLs that don't contain any active attempt to steal login credentials.

I repeat this process over the course of several days, always using the freshest suspect URLs, until I have about a hundred verified phishing URLs. Different types of phishing URLs are prevalent at different times. Some days all the products fare worse than others. Rather than report the raw percentages of URLs correctly blocked, I report the success of the product being tested relative to Norton . A value like "-9%" in the Norton column means that the product's score was nine percentage points lower than Norton's.

About Our Expert

Neil J. Rubenking

Principal Writer, Security

My Experience

When the IBM PC was new, I served as the president of the San Francisco PC User Group for three years. That’s how I met PCMag’s editorial team, who brought me on board in 1986. In the years since that fateful meeting, I’ve become PCMag’s expert on security, privacy, and identity protection, putting antivirus tools, security suites, and all kinds of security software through their paces.

Before my current security gig, I supplied PCMag readers with tips and solutions on using popular applications, operating systems, and programming languages in my "User to User" and "Ask Neil" columns, which began in 1990 and ran for almost 20 years. Along the way, I wrote more than 40 utility articles, as well as Delphi Programming for Dummies and six other books covering DOS, Windows, and programming. I also reviewed thousands of products of all kinds, ranging from early Sierra Online adventure games to AOL’s precursor Q-Link.

In the early 2000s, I turned my focus to security and the growing antivirus industry. After years of working with antivirus, I’m known throughout the security industry as an expert on evaluating antivirus tools. I serve as an advisory board member for the Anti-Malware Testing Standards Organization (AMTSO), an international nonprofit group dedicated to coordinating and improving testing of anti-malware solutions.

The Technology I Use

Much of the testing I do, particularly testing with real-world ransomware, is just plain dangerous. To perform such tests safely, I sequester them inside virtual machines managed by VMWare Workstation. For cross-platform testing, I use a MacBook Air, a Google Pixel 4, and a 6th-generation iPad.

I rely on my Delphi coding skills to create and maintain small applications. These include programs to check whether an antivirus correctly handled the malware it detected, launch dangerous URLs and record the security program’s reaction, and analyze the malware that I collect for use in testing. I also wrote a tiny browser and text editor for use in testing security apps that have predefined reactions for known products.

I do my writing and research on a Dell OptiPlex desktop, relying on Microsoft Word (my fingers know all the shortcuts). Many of my articles include charts and analysis; Excel is my go-to for those. When work hours end, though, I escape the bounds of Microsoft and Windows. There’s an iPhone in my pocket, I relax with my oversized iPad, and my Kindle Oasis is always loaded with the best science fiction and fantasy.

Read the latest from Neil J. Rubenking

Read full bio