Fix What Matters:Why CVSS Sucks and How To Do Better presented at BSidesDetroit 2014

by Michael Roytman,

Summary : Speakers Michael Roytman Converge Conference DetroitVulnerability prioritization is where everybody goes wrong and nobody knows it. CVSS currently guides the vulnerability management strategies for most organizations. This talk will prove why the standard is at best obsolete, by using analytical model analysis as well as live data from 20,000 organizations, over 1 million assets and over 50 million vulnerabilities. By correlating this dataset to a previously unexplored dataset of live, ongoing breach events, we analyze, for the first time with hard data, the effectiveness of vulnerability management strategies through sensitivity and predictive value positive analysis, and offer alternatives that fare far better than the status quo.
What we had to work with: 1. 23,000,000 live vulnerabilities across 1,000,000 real assets, which belong to 9,500 clients. 2. 1,500,000 real breaches which occurred between June and July of 2013 and were correlated to 103 CVEs. What we did: 1. Treated the two samples of breaches and vulnerabilities (incorrectly, but usefully) as coming from the same population. 2. Calculated the conditional probabilities of certain types of vulnerabilities being live-breached vulnerabilities. What we found: 1. The best policy was fixing vulnerabilities with entires in both Metasploit and Exploit DB, yielding about a 30% success rate, or 9x better than anything CVSS gets to, and 15x better than random. 2. Randomly picking vulnerabilities gives one about a 2% chance of remediating a truly critical (that is, one that has observed breaches in the past two months) vulnerability. 3. Randomly remediating a CVSS 10 vulnerability gives you a 3.5% chance of fixing a critical vulnerability. 4. If your policy is one of fixing vulnerabilities in Exploit DB, you have a 13% chance of remediating vulns with observed breaches. 5. Metasploit only? 25% chance.