Joseph Lee

KPI for Measuring Vulnerability Management Performance

Every business has mission critical activities. Security controls are meant to protect those critical activities to ensure business operations and strategic goals can be sustained indefinitely. Using an “Install and forget”-approach to security provides few assurances for achieving these objectives. An ever-changing digital landscape means a security gap could lead to a high stakes data breach. Things like privilege creep, server sprawl, and configuration errors tend to pop-up like weeds. Security teams who don’t continuously monitor don’t catch them – attackers do. For this reason, cyber security frameworks tend to be iterative processes that include monitoring, auditing, and continuous improvement.

Security officers should be asking: What does our organization need to measure to gain strong assurances and enable continuous improvement? In this article we will take you through a rationale for Key Performance Indicators (KPI) in cyber security outlined by industry leaders such as NIST and The SANS Institute and define a core set of vulnerability management specific KPIs. The most fundamental KPIs covered here can serve as a starting point for organizations implementing a vulnerability management program from scratch, while the more advanced measures can provide depth of visibility for organizations with mature vulnerability management programs already in place.

Cyber Security KPI Support Core Strategic Business Goals

KPI are generated by collecting and analyzing relevant performance data and are mainly used for two strategic goals. The first is to facilitate evidence-based decision making. For example, KPI can help managers benchmark how vulnerability management programs are performing in order to assess the overall level of risk mitigation and decide whether to allocate more resources or accept the status-quo. The second core strategic goal that KPIs support is to provide accountability of security activities. KPI can help identify causes of poor performance and provide an early warning of insufficient or poorly implemented security controls. With proper monitoring of vulnerability management performance, the effectiveness of existing procedures can be evaluated, allowing them to be adjusted or supplemented with additional controls. The evidence collected while generating KPIs can also be used to demonstrate compliance with internal policies, mandatory or voluntary cyber security standards, or any applicable laws and regulations by evidencing cyber security program activities.

The scope of measuring KPI can be enterprise-wide or focused on departments or infrastructure that is critical to business operations. This scope can also be adjusted as a cybersecurity program matures. During the initial stages of starting a vulnerability management, only basic information may be available to build KPI metrics from. However, as a program matures, data collection will become more robust, supporting more complex KPI metrics. More advanced measures may also be justified to gain high visibility for organizations with increased risk.

Types of Cyber Security Measures

NIST SP 800-55 V1 (and it’s predecessor NIST SP 800-55 r2) focuses on the development and collection of three types of measures:

  • Implementation Measures: These measure the execution of security policy and gauge the progress of implementation. Examples include: the total number of information systems scanned and the percentage of critical systems scanned for vulnerabilities.
  • Effectiveness/Efficiency Measures: These measure the results of security activities and monitor program-level and system-level processes. This can help gauge if security controls are implemented correctly, operating as intended, and producing a desirable outcome. For example, the percentage of all identified critical severity vulnerabilities that have been mitigated across all operationally critical infrastructure.
  • Impact Measures: These measure the business consequences of security activities such as cost savings, costs incurred by addressing security vulnerabilities, or other business related impacts of information security.

Important Indicators for Vulnerability Management

Since vulnerability management is fundamentally the process of identifying and remediating known vulnerabilities, KPI that provide insight into the detection and remediation of known threats are most appropriate. In addition to these two key areas, assessing a particular vulnerability management tool’s effectiveness for detecting vulnerabilities can help compare different products. Since these are the most logical ways to evaluate vulnerability management activities, our list has grouped KPI into these three categories. Tags are also added to each item indicating which purpose specified in NIST SP 800-55 the metric satisfies.

While not an exhaustive list, here are some key KPIs for vulnerability management:

Detection Performance Metrics

  • Scan Coverage (Implementation): This measures the percentage of an organization’s total assets that are being scanned for vulnerabilities. Scan coverage is especially relevant at the early stages of program implementation for setting targets and measuring the evolving maturity of the program. Scan coverage can also be used to identify gaps in an organization’s IT infrastructure that are not being scanned putting them at increased risk.
  • Mean Time to Detect (MTTD) (Efficiency): This measures the average time to detect vulnerabilities from when information is first published and when a security control is able to identify it. MTTD may be improved by adjusting the frequency of updating a vulnerability scanner’s modules or frequency of conducting scans.
  • Unidentified Vulnerabilities Ratio (Effectiveness): The ratio of vulnerabilities identified proactively through scans versus those discovered through breach or incident post-mortem analyses. A higher ratio suggests better proactive detection capabilities.
  • Automated Discovery Rate (Efficiency): This metric measures the percentage of vulnerabilities identified by automated tools versus manual discovery methods. Higher automation can lead to more consistent and faster detection.

Remediation Performance Metrics

  • Mean Time to Remediate (MTTR; Efficiency): This measures the average time taken to fix vulnerabilities after they are detected. By tracking remediation times organizations can gauge their responsiveness to security threats and evaluate the risk posed by exposure time. A shorter MTTR generally indicates a more agile security operation.
  • Remediation Coverage (Effectiveness): This metric represents the proportion of detected vulnerabilities that have been successfully remediated and serves as a critical indicator of effectiveness in addressing identified security risks. Remediation coverage can be adjusted to specifically reflect the rate of closing critical or high severity security gaps. By focusing on the most dangerous vulnerabilities first, security teams can more effectively minimize risk exposure.
  • Risk Score Reduction (Impact): This metric reflects the overall impact that vulnerability management activities are having to risk. By monitoring changes in the risk score, managers can evaluate how well the threat posed by exposed vulnerabilities is being managed. Risk Score Reduction is typically calculated using risk assessment tools that provide a contextual view of each organization’s unique IT infrastructure and risk profile.
  • Rate Of Compliance (Impact): This metric represents the percentage of systems that comply with specific cyber security regulations, standards, or internal policies. It serves as an essential measure for gauging compliance status and provides evidence of this status to various stakeholders. It also serves as a warning if compliance requirements are not being satisfied, thereby reducing the risk of penalties and ensuring the intended security posture put forth by the compliance target.
  • Vulnerability Reopen Rate (Efficiency): This metric measures the percentage of vulnerabilities that are reopened after being marked as resolved. Reopen rate indicates the efficiency of remediation efforts. Ideally, once a remediation ticket has been closed, the vulnerability does not issue another ticket.
  • Cost of Remediation (Impact): This metric measures the total cost associated with fixing detected vulnerabilities, encompassing both direct and indirect expenses. Cost analysis can aid decisions for budgeting and resource allocation by tracking the amount of time and resources required to detect and apply remediation.

Vulnerability Scanner Effectiveness Metrics

  • True Positive Detection Rate (Effectiveness): This measures the percentage of vulnerabilities that can be accurately detected by a particular tool. True positive detection rate measures the effective coverage of a vulnerability scanning tool and allows two vulnerability scanning products to be compared according to their relative value.
  • False Positive Detection Rate (Effectiveness): This metric measures the frequency at which a tool incorrectly identifies non-existent vulnerabilities as being present. This can lead to wasted resources and effort. False positive detection rate can gauge the reliability of a vulnerability scanning tool to ensure it aligns with operational requirements.

Key Takeaways

By generating and analyzing Key Performance Indicators (KPIs), organizations can satisfy fundamental cybersecurity requirements for continuous monitoring and improvement. KPI also supports core business strategies such as evidence-based decision making and accountability.

With quantitative insight into vulnerability management processes, organizations can better gauge their progress and more accurately evaluate their cyber security risk posture. By aggregating an appropriate set of KPIs, organizations can track the maturity of their vulnerability management activities, identify gaps in controls, policies, and procedures that limit the effectiveness and efficiency of their vulnerability remediation, and ensure alignment with compliance with internal risk requirements and relevant security standards, laws and regulations.

References

National Institute of Standards and Technology. Measurement Guide for Information Security: Volume 1 — Identifying and Selecting Measures. NIST, January 2024, https://csrc.nist.gov/pubs/sp/800/55/v1/ipd

National Institute of Standards and Technology. Performance Measurement Guide for Information Security, Revision 2. NIST, November 2022, https://csrc.nist.gov/pubs/sp/800/55/r2/iwd

National Institute of Standards and Technology. Assessing Security and Privacy Controls in Information Systems and Organizations Revision 5. NIST, January 2022, https://csrc.nist.gov/pubs/sp/800/53/a/r5/final

National Institute of Standards and Technology. Guide for Conducting Risk Assessments Revision 1. NIST, September 2012, https://csrc.nist.gov/pubs/sp/800/30/r1/final

National Institute of Standards and Technology. Guide to Enterprise Patch Management Planning: Preventive Maintenance for Technology Revision 4. NIST, April 2022, https://csrc.nist.gov/pubs/sp/800/40/r4/final

SANS Institute. A SANS 2021 Report: Making Visibility Definable and Measurable. SANS Institute, June 2021, https://www.sans.org/webcasts/2021-report-making-visibility-definable-measurable-119120/

SANS Institute. A Guide to Security Metrics. SANS Institute, June 2006, https://www.sans.org/white-papers/55/

Contact Free Trial Buy Here Back to Overview

/by