The most common test for an antivirus utility involves showing it thousands or millions of samples and checking how many it recognizes. More advanced tests observe the product in action as it cleans up malware infestations and prevents attacks. Vulnerability researcher Tavis Ormandy feels this just isn't the right way to evaluate a product.
In a talk at the Black Hat conference Ormandy, better known as a Google infosec engineer, reported on a completely different approach. Ormandy contends that antivirus vendors should be completely open about the way their products work, rather than hiding behind market-speak and techno-babble.
Some vendors say that revealing their techniques and algorithms would weaken their security. Ormandy referenced Kerckhoff's principle, which states that a cryptographic system should be secure even if everything about it is public. He argues that the same should be true of antivirus software.
Ormandy chose Sophos, a well-known enterprise-focused antivirus, for an experiment. Working from a list of stated features, Ormandy reverse-engineered and disassembled the product to see exactly how those features are implemented. The results were alarming.
Putting Sophos to the Test
Virus signatures are the simplest part of any antivirus product, so Ormandy decoded the system used to store Sophos's signatures. He concluded that despite claims that the signatures are reviewed by expert researchers, the vast majority are auto-generated and many reference irrelevant data.
The product offers exploit mitigation, but Ormandy determined that it does absolutely nothing under Windows versions prior to Vista. It still loads and links to every running process under Vista or later, but it does nothing. Based on other elements of this component he concluded that whoever wrote the code for this feature misunderstood certain security aspects of Windows.
Ormandy reverse-engineered the proprietary 64-bit encryption system used in Sophos's antivirus, not to be confused with Sophos SafeGuard full disk encryptoin. Modern encryption systems generally use keys of 256-bits or more. He found that Sophos's system requires the decryption key to be present in the file, and concluded that it's not really encryption at all, just a layer of obfuscation to slow down hackers.
The product's pre-execution analysis feature runs suspect code in an emulator to detect dangerous activities. However, it's barely functional. Sophos touts the ability to unpack malware hidden using packers, but actually it doesn't work with any modern packers. The "gene and genotype" system is extremely simplistic. And so on.
This presentation was certainly an eye-opener. Ormandy declined to speculate whether other prominent antivirus products would have fared any better. He reported that Sophos was "good-natured and receptive" when he sent them a draft of the presentation, and that they're already working to fix some of the problems he reported. I hope he'll put more products through this grueling regimen in the future.