Game-theoretic framework to assess attack-resistance of an AV system presented at Virus Bulletin 2008

by Bin Mai (Northwestern state university),


Summary : "Malware authors are continuously probing an anti-virus system (AVS) for its vulnerabilities and developing new stealth
mechanisms to take advantage of those vulnerabilities. We present a game-theoretic framework to model the strength of an
AVS against such evolving offences. Game theory provides the right structure for such an analysis because it can account
for both the accuracy of individual components of an AVS and also the cost of developing stealth mechanisms that take
advantage of the AVS's weaknesses. The framework presented enables analytic evaluation of an AVS, and thus paves the
way for the design of an optimal AVS.
The framework treats an AVS as a composition of special-case detectors (SCDs) such as MD5 checkers, X-ray scanners,
heuristic behaviour-matching dynamic code emulators, etc. The composition is by means of selector logic that determines
which SCDs are invoked on a given sample. By attaching costs and pay-offs for the attacker and defender, game-theoretic
analysis can be performed. Using this framework we show that the compositions are beneficial only when the cost of
developing stealth techniques is above certain model thresholds. We also show that, surprisingly, when stealth design
is easy and selector accuracy is high, the difference in detection rates of the SCDs should be low for optimal
performance of the AVS.