Demorpheus: Getting Rid Of Polymorphic Shellcodes In Your Network presented at DEF CON 20

by Dennis Gamayunov, Svetlana Gaivoronski,

Summary : One of the most effective techniques used in
CTF is the usage of various exploits, written
with the help of well-known tools or even
manually during the game. Experience in CTF
participation shows that the mechanism for
detecting such exploits is able to significantly
increase the defense level of the team.
In this presentation we propose an approach
and hybrid shellcode detection method, aimed
at early detection and filtering of unknown 0-day
exploits at the network level. The proposed
approach allows us to summarize capabilities
of shellcode detection algorithms developed
over recent ten years into optimal classifiers.
The proposed approach allows us to reduce the
total fp rate almost to 0, provides full coverage of
shellcode classes detected by individual classifiers
and significantly increases total throughput of
detectors. Evaluation with shellcode datasets,
including Metasploit Framework 4.3 plain-text,
encrypted and obfuscated shellcodes, benign
Win32 and Linux ELF executables, random data
and multimedia shows that hybrid data-flow
classifier significantly boosts analysis throughput
for benign data up to 45 times faster than
linear combination of classifiers, and almost
1.5 times faster for shellcode only datasets.