CISO Says: Patch or accept?

I remember a quote from a 2013 Darkreading article. The quote is from Eric Byres, at that time, the CTO of Belden's Tofino Security, and I believe it is a central problem in maintenance and security:

It's completely a career-limiting move for an engineer if he installs a patch to fix something that isn't obviously broken, and it ends up shutting down the plant.

Although this quote was about industrial control systems, it is valid for any system in which uptime is vital for the organization. So patching is not without risk, but not patching comes with a risk too. When does the coin flip from "don't touch" to "patch"?

The impact and the likeliness determine the risk of any non-patched system. The overall risk is usually "calculated" by simple multiplication. I have argued before (see my Lies! Blog series) there are more sophisticated ways to calculate the risk, but still increasing impact will result in increased risk. Similarly, increasing likeliness will result in increased risk as well. We can then compare this with the cost of mitigating the risk by applying a patch.

What does patching cost?

As an exercise, let's assume a small company uses ten different products (like a Windows OS, Database, ERP system, etc.), which vendors all supply a monthly update. This will result in 120 patches per year to validate, test, and apply. This will cost roughly three working days per patch at a rate of 400 Euros per day. Every now and then an update will result in some damage in the form of an outage. Let's assume an outage happens once in every 50 applied patches and that the resulting damage is in 90% of the cases between 2000 and 20.000 Euro.

• Cost of applying patches = 120 x 3 x 400 = 144.000 Euro
• Risk applying patches = 19.400 Euro (for calculation see CISO says Lies!, part 2)

Of course, this is a simplified calculation. There may be other costs involved, such as the planned downtime needed to apply the patch. However, after careful analysis, we can reasonably predict what the price will be. In this example, it is almost 2 FTE!

The calculation is optimistic. Once you deploy a vulnerability, you may find hundreds of unpatched vulnerabilities. It is not uncommon to find 500-600 vulnerabilities when scanning 100 IP addresses. The cost of mitigating all these vulnerabilities with patches or other means will be high, and it will bring a severe load on your IT workforce.

What does not patching cost?

Not patching increases the risk by an unpredictable amount. Depending on what problem is fixed by the patch, the risk may increase slightly or may increase significantly. 

In the example above the costs are around 165k Euro. Risks that will result in an expected loss around this amount are e.g.

The first row in this table might be the result of a vulnerability found in a system exposed to the internet. An example was the Pulse Secure problem in 2019, which allowed attackers to gain access inside private VPN networks. The likeliness in the table is 20%, which is a low estimation. Organizations of interest to nation-states will have a nearly 100% chance of being attacked in cases like this, but they may not be the only ones. 

What should you do? Patch or not?

Patching everything is not feasible. Not patching at all is very risky. We must decide:

1. What to patch because not patching will impose an unacceptable risk to the organization.
2. What not to patch because problems and cost will outweigh the benefits, and if the patch causes issues, it will be a career-limiting move.

There is no golden rule on how to decide, but as always, we should evaluate impact and likeliness. The following questions will help with the evaluation:

1. What would be the impact if the system becomes compromised?
   • Will the business stop?
   • Will there be a data breach?
   • Will the attacker gain access to other systems?
   • Could the attacker encrypt all the data?
2. Is there a work-around/plan B available?
3.  How likely is it a problem will occur?
   • Does the system have known vulnerabilities?
   • Can these vulnerabilities be exploited?
4. How easy can an attacker reach the system?
   • Is there an exploit available? Is it already being used?
   • Are there any mitigating measures in place?
5. What is the risk of patching?
   • Does the supplier allow patching?
   • Can we test the patch before applying it in production?
   • How easy is it to revert the patch?

Not all these questions can be easily answered in all situations, and not every answer will be a simple yes/no. It is best illustrated with a few examples.

VPN authentication flaw:

In 2019 a flaw was found in a VPN solution. This flaw in multiple versions of Pulse Connect Secure and Pulse Policy Secure gave remote attackers a way to connect via HTTPS to an enterprise network without needing any valid username or password. The impact was severe in most cases because it allowed attackers access to the heart of the internal network. The likeliness of an attack was very high because an exploit was freely available, and there were many reported attacks in a short time. This type of vulnerabilities should be patched without delay.

Docker denial of service:

An example on the other side of the spectrum is the Docker problem found in August 2019, which in theory would allow an attacker to make Docker crash or run programs as your login. However, this only allowed a local attacker to cause a denial of service. The likeliness is very low. And there is no proof of concept available that you can execute a program. This type of vulnerability can be patched during regular maintenance.

Chain of vulnerabilities:

Another problem is the fact that sometimes the combination of small problems can result in a big problem. This is often called chain vulnerabilities. You will find some good examples here.

Structured approach

So, how do we decide on treating each vulnerability? In the table below, we identified three system priorities. The priority reflects  

1. the impact of a breach of the system on the organization and 
2. the likeliness that this system will be exploited once a vulnerability exists.

The priority will be High for systems containing sensitive data (high risk when disclosed), systems the company is dependent on (the website of a webshop), and systems exposed to the internet (like the VPN in the example above).

The vulnerabilities are classified as well, depending on the impact and likeliness. The impact is defined by what action the vulnerability allows an attacker to perform. Can the attacker gain unauthorized access, or can they perform a denial-of-service? The and likeliness of misuse is defined by how much effort is needed to perform the attack. Is there an exploit available? Do we have other protective measures in place?

Unfortunately, you cannot mitigate everything. Fortunately, in many cases, the vulnerability imposes such a small risk it can be accepted. You can find statistics on the average severity of vulnerabilities published at CVE details. Many times, you can actually accept the risk and not patch in case of low-priority systems or low-severity vulnerabilities. However, you should be careful if there's a possibility to combine several vulnerabilities.

Some of the vulnerabilities, time is critical, and you need to act immediately. An example was the Citrix vulnerability in 2019, where an exploit was available before the patch. For these cases, becoming aware of vulnerabilities and detecting the existence in your organization should be automated, and the vulnerability should be treated as an incident. 

Most of the vulnerabilities that are not accepted should be solved in the change process. In that way, we ensure that applying a patch or changing a configuration will not introduce new and unnecessary risks. After all, vulnerability management is about reducing risk!

Want to learn more about managing cybersecurity risk and costs? Download our whitepaper, The ROI of Cybersecurity.