Historically, there have been two schools of thought about patching: Patch as soon as a patch is available to eliminate security risks versus testing the patch extensively before it is applied to production systems ( aka “Never change a running system” ). One of the main reasons why you might feel the need to patch are vulnerabilities. These come as CVEs.
What is a CVE ?
Once a vulnerability is known, it is assigned an identifier in the Common Vulnerability and Exposures database (CVE). A typical identifier e.g. is CVE–2017-5638, telling us that the vulnerability was detected in 2017. These identifiers help to find and reference these vulnerabilities in security-related tools.
Usually, a vulnerability is reported and the vendor or distributor is given some time to fix the issue (see https://en.wikipedia.org/wiki/Coordinated_vulnerability_disclosure). Once a patched version of the software is available, the CVE is made public.
Keeping production secure
Once a CVE is public, you need to know a few things:
a) Which systems are affected ?
b) How long will it take to patch production ?
c) When can I start to patch production ?
d) What are the risks involved ?
e) Are there other remedies (mitigation) ?
All important questions. Some are related to the DORA Metrics (see https://open011prod.wpengine.com/2022/10/31/devops-metrics-what-are-they-good-for/ ). It essentially boils down to the ability to make changes and getting those changes into production without breaking anything.
Facts
Going back to the initial question, let’s have a look at some numbers:
– the top third of vulnerabilities exploited in the wild have a severity of 9 or higher ( a severity of 9 or higher basically meaning that immediate action is required ), see https://weis2019.econinfosec.org/wp-content/uploads/sites/6/2019/05/WEIS_2019_paper_53.pdf
– only half of the exploits have publicly available information
– 18 0-day exploits have been found in the first half of this year alone, see https://googleprojectzero.blogspot.com/2022/06/2022-0-day-in-wild-exploitationso-far.html ; these are often variants of previous exploits that were not properly fixed
– on average, it takes 24 hours until a published vulnerability is exploited, see https://vulners.com/stats
– known exploits typically are seen in the wild for 359 days
One example is the Equifax security breach. Hackers exploited CVE–2017-5638 ( Apache Struts ). The CVE was published in March of 2017, with the exploit taking place in March 2017 and again in May/July 2017. The vulnerability supposedly was patched, but no one had tested whether the patch actually was applied. The attackers could roam free in the Equifax network for an extended period of time ( 134 days ) and steal sensitive information of more than 140 million people in the US, UK and Canada including full name, credit card information and social security number.
One positive example of patching done right is the way that IKEA handled ShellShock. In a nutshell:
– just two people
– patched more than 3500 servers on over 400 sites in 35 different countries on 4 continents
– in mere hours
See for yourself at https://www.redhat.com/en/about/videos/ikea-vs-shellshock or just the video at https://www.youtube.com/watch?v=aZA1JHMcd6I
What does this mean ?
– If you don’t react (either through patching or other forms of mitigation) within 24 hours of a vulnerability becoming public, your exploitation risk is above 50%.
– A lot of companies are either not patching at all (“Never change a running system”) or are very slow at patching
– It is possible to patch a large number of systems in a very short time frame, it just takes a lot of preparation before bad things happen
– Patching is just one of the tools in your belt, and you should verify that a patch is actually applied. Other tools that help mitigate vulnerabilities are e.g. Web Application Firewalls, Intrusion Detection Systems, Intrusion Prevention Systems and so on
– Monitoring (and alerting if something out of the ordinary happens) will help to minimize the duration and impact of an attack
– Architecting your network for Zero Trust ( for an explanation see https://en.wikipedia.org/wiki/Zero_trust_security_model ) will help, as it protects your most valuable assets and limits the attack surface.
But what about containers ?
Actually, patching for containers works in a totally different fashion:
- Ideally, once a fix is known, you rebuild the affected container images automatically with the help of your CI/CD pipeline
- After having done that, you can just refresh your containers enjoying a seamless patch experience
Conclusion
Ignoring known vulnerabilities (aka “Never change a running system”) is not really an option anymore. Taking several weeks to patch systems in production is getting riskier all the time. You need to weigh the risk of running production with known vulnerabilities against the risk of patching production. The more often you deploy into production on a regular basis, the less risky it is to patch your software. Your ultimate goal should be to make your regular deployment process so fast that you don’t need a separate process for patching.