You Can't Filter "The Stupid" presented at OWASP AppSecAsiaPac 2012

by Charles Henderson, Daniel Crowley,

Summary : Everyone wants to stretch their security budget as far as possible; in recent years, automated application security tools have become a popular choice for doing so. However, manual security testing isnt going anywhere until the HAL-9000 application scanner/web app firewall comes online. While automated tools may be tempting, the reality is that only manual application testing provides strong protection against modern threats. Companies that are serious about application security and have reviewed both options are consistently choosing manual testing.
Logic flaws may not get the press that vulnerabilities like SQL Injection or Cross-Site Scripting (XSS) do, but they can be devastating to an application. Every application is going to have its own unique set of logic, so it is impossible to automate tests for logic vulnerabilities. Because logic flaws often require no hacking skills, standard users often discover the vulnerabilities on their own. Examples from Trustwave penetration tests range from the simple such as a shopping cart application that accepts bogus coupon codes to the very complex sensitive information disclosure by combining query results across multiple systems.
Many vulnerabilities are simply too complicated to practically detect with an automated tool. For example, it is very common for web applications to provide complex data structures such as serialized objects to the web browser. Examples of such frameworks or techniques include Microsofts .Net, Java ServerFaces, JSON, and Adobe Flex. Since a developer can place any type of data in these structures, an automated tool cannot be expected to reliably test them. Analyzing these structures can be a very complex process that requires the ability to understand the data in the context of the application.
An experienced penetration tester can identify complicated vulnerabilities in the same way that a human attacker does. Humans can comprehend the intention of the developer in how the application is designed and intended to operate. Understanding this is critical for identifying how the system can be subverted. Human testers can also deduce business logic rules, even if they are not explicitly documented. When business requirements are documented and provided to the tester, the quality of testing is even greater.
Manual source code reviews present even more benefits by identifying vulnerabilities that require access to source code. Examples include hidden or unused application components, which may have been left intentionally as backdoors by disgruntled developers. There are many forms of blind SQL injection with no evidence in the response, exotic injection attacks (e.g. mainframe session attacks), vulnerabilities in back-end systems, and intentional backdoors.

Daniel Crowley: Daniel does pen testing, research, training, and various other things for Core Security Technologies. In his spare time, he plays around mostly with Web-based technologies and locks. Being an entertainer by nature, Daniel likes combining art with technology and his presentations are designed to inform AND entertain. Daniel was a speaker at Shmoocon VI and won the Gringo Warrior competition at Shmoocon V.