Effectively Testing APT Defences presented at AVAR 2014

by Richard Ford, Gabor Szappanos, Simon Edwards,

Summary : Anyone watching the cybersecurity marketplace will have noticed a rapid rise in products that claim to provide protection from “Advanced Persistent Threats” (APTs). As targeted attack get more attention, and protection products pay more attention to the implementation of new defensive technologies, the need arises for the testing of the products specific to this new kind of threat. However, compared to general product testing, APT presents additional challenges for the testers. In this presentation, we ask if APT protection can be tested, and if so, can it be done practically.
Under the umbrella of the Anti-Malware Testing Standards Organization (AMTSO), we have started working out the best practices for this important field. However, several of the characteristics of APTs make application of AMTSO’s best practice testing guidelines difficult.
A fundamental premise of AMTSO’s philosophy is that good tests try to reproduce an attack in a real-life setting, with the aim of recording the protective measures employed by the security software to stop the attack at the earliest stage. The ultimate goal is to assess how well the solution protected the victim system from attack. However, as an APT represents not a single point in time, but a coordinated and often spread over a long time, any test should reflect this. However, this poses significant issues with respect to sample selection and verification.
Even if defendable samples could be obtained, measuring on-demand detection rates of the installed APT backdoors is a trivial but bad choice for this kind of test, missing both the time history of the attack and the depth of the defence. In this presentation, we explore more fully why this is a bad test, and provide some guidelines on how APT test might be performed properly.
As an outline, this presentation will explain the general timeline of an APT attack, identifies the different stages of the attack and the malware defence components that are effective in blocking the attack at the particular stage, and proposes workable testing strategies that can be employed at each step to measure the efficacy of APT defences.