by Seokwoo Choi,

Summary : API wrapping technique is commonly used among malwares and code obfuscators. One of the advanced API wrapping techniques is runtime-obfuscation used by Themida. Runtime-obfuscation makes it difficult to reverse engineer by obfuscating API function on each run. So far, binary pattern matching or pattern-based code optimization techniques have been used to identify the original API functions from the runtime-obfuscated functions. Applying a new obfuscation pattern easily breaks these pattern-based approaches.
In this talk, I present a more resilient API deobfuscation scheme based on memory access analysis. This method utilizes memory access pattern of runtime-obfuscation technique. The embedded runtime-obfuscator in the packed binary obfuscates one API function at a time. While an API function is obfuscated, each memory value of the instructions is read, the instruction is transformed into obfuscated instructions, and the obfuscated instructions are written into a newly allocated memory block. Thus, a set of memory write addresses during one API function obfuscation process is a superset of the corresponding obfuscated API function addresses. API deobfuscator is implemented based on the explained memory access analysis. Every memory write address is recorded after an API function is read from memory before the next API function is read. After each API function is obfuscated, a map from each API function into obfuscated function addresses is constructed. The original API functions are identified by applying the target addresses of obfuscated function calls to the map.
The API deobfuscator utilizes Intel Pin to track memory accesses. The tool executes Themida packed binary files until the original entry point and restores every obfuscated API function call to the original API function call. The deobfuscated process can be analyzed by common debuggers such as Ollydbg.