Simple Entropy-Based Heuristics For Log And Traffic Analysis presented at ShmooCon 2007

by Sergey Bratus,

Tags: Security Analysis

Summary : I argue that introducing entropy-based features to log and traffic analysis tools allows the admins to quickly notice otherwise hidden anomalies and organize the data in ways that best show off the overall structure and peculiarities of each input data set.
Entropy and related information measures provide a way to describe the overall shape of data distributions in logs. This makes it easier to notice anomalous values, to cluster and summarize records for convenient browsing, and to notice correlations that may be hard to find otherwise. For large logs, it is easy to get lost scrolling down many screens of records; with entropic measures one can get the general idea of the composition of a data set and the most likely places to look for an anomaly. Together, these simple heuristics can significantly speed up log analysis. I will show of a prototype log viewing tool that incorporates them.
To demonstrate this approach for packet data, several new panes and a number of new functions are introduced into Ethereal (to be demoed).