Play by the rules? Should AV be enforcing the rules to prevent uncontrolled obfuscation by malware? presented at Virus Bulletin 2010

by Rachit Mathur (McAfee),

URL : http://www.virusbtn.com/conference/vb2010/abstracts/MathurKapoor.xml

Summary : This paper will present the most up-to-date techniques used by malware to hide in the crowd, e.g. using innocent-looking
code or masquerading as a legitimate packer, MSVC file or corrupt file, etc. We will discuss smart universal rules that can
be applied by any AV to block malware from using these techniques. For example, such rules can be as simple as blocking
the use of the .reloc section name for anything other than relocations. More complex rules may include blocking all files
that use call obfuscations except for a few known packers, or preventing checks on the process default heap header except
Themida, etc. To counter the upsurge of these masquerading malware we have been enforcing such policies by looking at
millions of clean applications and malware. We will show how these detection rules have helped McAfee change the landscape
of obfuscation techniques - even forcing some of these techniques to become obsolete in the wild, thus limiting the
playground for malware.
This presentation will also discuss our experiences with building file reputations using these rules to enforce separation
of malicious files from clean applications. Finally, we discuss the question of how far we can and should go in enforcing
such rules. Are these justified or do they encroach on the right to freedom of programming?