Threat Hunting Techniques at Scale presented at FirstConferenceKualaLumpur 2018

by Dhia Mahjoub, Thomas Mathew,

Summary : Threat hunting is an important process in every security operation, whether it is meant to produce intelligence for internal or external use it consists in proactively searching through large scale network data to detect and pinpoint threats that evade automated and signature-based security systems. In today’s talk, we discuss the different steps of efficient threat hunting at scale: we describe how to initially use a set of short term high signal seeds from manual analysis to uncover additional threats (domains, IPs, binaries, etc). Then, we introduce a set of techniques that facilitate the automated generation of long term signals associated with the detection of malicious campaigns (botnets, malspam, ransomware). The generation of these signals involves analyzing vast quantities of hourly global DNS query traffic to identify patterns that exhibit non-random anomalous behaviour. These signals have proven to have long term predictive power because they model the network effects of a campaign as it spreads globally. Specifically, network signals are more difficult for a malicious operator to obfuscate and thus these signals can be used for an extended period of time. Generating these signals depends on having large amounts of DNS data to statistically ensure that the anomalies detected can be considered non-random. We show how the anomalies arising in DNS query patterns, SSL hosting infrastructures, and client lookups can all be used to generate a set of initial domains or IPs that can be further researched. By correlating similar hosting patterns between such domains we can identify malicious campaigns. When it came to generating a seed list from SSL data we used a graph-based approach that identified anomalous subgraphs within the global SSL hosting infrastructure which lead us to uncover patterns of criminal hosting space that leverages SSL. Subsequently, we show the importance of investigating overarching patterns and TTPs behind malicious campaigns in order to go beyond short-lived IOCs and develop an understanding of the operational setup of criminal actors. This can provide us a proactive and longer-lasting advantage over the adversary. Our talk will not only go over the statistical methods used to identify these anomalies but also describe the details of the backend infrastructure required to allow for the quick detection of these threats.