Tutorial 2: Empirical Validation of Vulnerability Discovery Models presented at ESSoS 2014

by Fabio Massacci,

Summary : A Vulnerability Discovery Model estimates the number of vulnerabilities that will be reported after a release by analysing past known vulnerability data. Accurate models can be useful instruments for both software vendors and users to understand security trends, plan patching schedules, decide updates, and forecast security investments. Thus far, several models have been proposed in the literature (e.g. by Anderson, Rescorla, Alhazami & Malaya etc.), not all of them are equally accurate or have been appropriately validated.
This tutorial will introduce attendees to an empirical methodology for the validation of such models but can also be very useful for anybody interested in empirical models based on public vulnerability data.
At first we will discuss a number of issues that might bias your study (and actually did indeed bias previous study) to set up your validation experiment. We will start from what actually is a vulnerability and how to count them, what is a piece of software, then we will discuss what is an experimental interval to fit the data and an appropriate statistical test, and finally what is an appropriate quality measures.
We will illustrate all these issues with concrete data from several years of data collection of our vulnerability studies across 4 browsers (Firefox, Chrome, Safari and IExplorer).