Rootkit Detection Via Kernel Code Tunneling presented at Blackhat Europe 2011

by Mihai Chiriac (Bit Defender),

Tags: Security

Summary : We present a novel rootkit detection technique called "kernel code tunneling". The technique uses a custom-made dynamic instrumentation framework to analyze execution flow.
While similar dynamic instrumentation engines do exist (e.g. Intel PIN), our engine offers significant advantages:
- it was designed for kernel mode operation
- it was designed to correctly handle potentially offensive code
Current rootkit detection engines either use methods like "cross view", or analyze specific data areas (e.g. IDT, SSDT) or code areas (e.g. they search for inline patches). However, rootkits are getting more and more complex. No more are inline patches limited to the first bytes of a function: we can now find them anywhere in the execution flow. Instead of a simple JMP/CALL to the malicious code, complex control transfer trampolines are now commonplace.
Our presentation will cover the following topics:
- design of a kernel-based dynamic instrumentation engine
- overcoming kernel-specific issues (IRQ levels, async tasks, self modifying code)
- analysis of various tunneling sessions, with/without active rootkits
- specific cases when instrumentation has provided us with enough data to effectively *clean* the machine