My article last week, How To Measure’s CISA’s Performance, was only out for hours before CISA published their first metrics for the FY 2024 – 2026 Strategic Plan.
The metrics are related to CISA’s Vulnerability Scanning Service. The key facts about this service are:
- Companies must sign up for the service. It’s opt in. CISA does not select critical infrastructure companies to scan. The metrics are likely not a representative sample of the US footprint or any specific sector.
- CISA only scans the Internet accessible attack surface in the subnets the opt in companies provided.
- The scanning is to identify if the Internet accessible attack surface has Known Exploitable Vulnerabilities (KEVs), is running vulnerable services (such as FTP or Telnet, but not vulnerable ICS services such as Modbus TCP or DNP3), and has certain configuration vulnerabilities.
- A similar, or even more comprehensive, service is available commercially from many product or service vendors.
- The service had 3,678 organizations subscribing in April 2022, and it had 5,907 in June of 2023.
There are two charts that contain the data. Below is the chart that shows the average number or KEVs in a company that are accessible from the Internet over time.
The trend line is in the right direction. We can’t tell the distribution of the KEVs across the organizations. Do a small number of companies have most of the KEVs? Or are the newer KEVs spread across a broader number of entities?
Being able to ask and answer these questions in the data set is where the value is to CISA. Companies can and should be doing this themselves, but then CISA wouldn’t get the data.
The metric I’d want most from this data is the time to remediate a new KEV. CISA is scanning at least weekly so they have this data.
This second chart identifies the change in Internet accessible vulnerable services. It appears that the scanning service has little impact on this issue. This is a useful data point that could be used to launch an awareness campaign or other activity if CISA views reducing Internet accessible vulnerable services as a priority.
Calling Foul / Misleading Headline And Writeup
This positive effort of providing a service and tracking and publishing metrics is dampened by the odd and misleading way it was written up.
The title of the article is: Cybersecurity Performance Goals: Assessing How CPGs Help Organizations Reduce Cyber Risk. There is no connection between the publishing of the CPGs and the results from the Vulnerability Scanning Service. If the CPGs never existed the results would be similar.
The slim reed they use to make this connection is the average number of KEVs per entity decreased slightly more and more linearly over a five month period (from .47 to .37 average KEVs per organization) after the CPGs were published.
The rate of enrollment in the Vulnerability Scanning Service is mostly consistent, a straight-ish upward sloping line before and after the CPGs were announced.
Conclusion
This Vulnerability Scanning Service is a smart move by CISA because it gets them data, even if it is only a sample of the Internet exposed environment. Publishing this data is great for the community to know the status and also for benchmarking. I’d prefer they perform this Internet scanning of key sectors regardless of opt in. The data would be even more valuable.
Using the Vulnerability Scanning Data to cheerlead for the impact of the CPGs is inaccurate and a mistake … unless like the Shields Up Campaign the CISA marketing machine convinces the Congress and press it’s the truth.