Analysis of an undocumented network protocol presented at HackLu 2008

by Jean-baptiste Bdrune,

Summary : This presentation deals with a methodology used during the analysis of an undocumented and encrypted protocol. In such a situation, it is rather hard to asset the security of a software since it is usually really big (lots of files), undocumented (no public specifications for the protocol) and the developers are not willing to cooperate. Whereas programming mistakes are "easy" to spot and patch, design flaws are much more difficult to notice and almost impossible to fix. Most of the time, these flaws are a bigger problem as they mean the behavior of the software can be tricked to perform deadly operations, not expected by the developers. Such flaws become really valuable nowadays, as it is quite difficult to exploit programming flaws thanks to defenses provided by the operating systems (randomization, NX bit, canary, and so on). Due to the amount of code present in such softwares, not all the client and server features could be examined: in the example we will detail, the server is composed of more than 200 binaries, for a total of 240Mb. We explain how we quickly sorted them to focus on interesting parts. Big softwares often have a debug mode that helps developers. This mode, documented or not, will help our analysis. One of our goals is to be able to speak the same protocol as the original software. In this way, we might be able to perform actions not expected by either the client or the server. So, packets exchanged between the server and the clients have been analyzed, and enough information has been retrieved to guess a basic packet format. A new client has then been developed, and has been modified all along the analysis process when new features were discovered, or errors were fixed. As a focus, we will show how we reverse engineered the authentication routines and some cryptographic algorithms. During the protocol comprehension, several classic vulnerabilities have been discovered, leading to denials of service or code execution. More interesting, a design error which allows any authenticated user to take a full control of the server will be explained. This error comes from the design of the server architecture. Its exploitation uses only the features provided by the protocol. As a conclusion, a demo based on a homemade Trojan will be shown.