iSnoop: How to Steal Secrets From Touchscreen Devices presented at BlackHatAbuDhabi 2011

by Stefano Zanero, Federico Maggi, Alberto Volpatto,

Summary : Spying on a person is an easy and effective method to obtain sensitive informations, even when the victim is well protected against common digital attacks. Modern mobile devices allow people to perform some information sensitive actions in unsafe places, where anyone could easily observe the victim while typing. What if your mobile phone has a cool touchscreen interface that gives you graphical feedback as you type (iPhone, Android, BlackBerry Torch)? Does it make shoulder surfing easier or, worse, automatable?
We believe so, and to demonstrate it, we developed a practical shoulder surfing attack that automatically reconstructs the sequence of keystrokes by aiming a camera at the target touchscreen while the victim is typing. Our attack exploits feedback such as magnified keys, often appearing in predictable positions. This feedback mechanism has been adopted by the top three touchscreen vendors (Apple iOS, Google Android, RIM BlackBerry); in newer version of these mobile OSs, the user has no way to disable it. To demonstrate the effectiveness of our approach, we implemented it against the iPhone (the most popular one), but it can be easily adapted to similar devices with minor modifications.
Our attack takes into account that, in real-world scenarios, both the victim's device and attacker's spying camera are not standing in fixed positions. To compensate their movements and misalignments, our system detects and rectifies the target screen before identifying keystokes. By doing that, we are able to automatically recognize up to 97.07% of the keystrokes, with as low as 1.15% errors and an average processing speed that makes it a fast and quasi-real-time alternative to shoulder surfing.