The RHEL root trust store and why it matters for security

Photo by Robynne Hu on Unsplash

In the perilous realm of digital security, trust is fragile. Transport Layer Security (TLS) web server authentication relies on trust anchors, but a flaw in the design puts web entities at risk of malicious impersonation. This article explores the origin of certificates in the RHEL root trust store and sheds light on the TrustCor case, highlighting the importance of removing compromised and unnecessary certificate authorities. Join us on a captivating journey through the intricate web of certificates, where the balance between trust and vulnerability is revealed. Brace yourself for a compelling narrative that uncovers the inner workings of the PKI system and its impact on our online security.

While trust is a fundamental building block of robust relationships, organizations, and societies, it is not immune to challenges and failures. In the world of digital security, trust is vital but often questioned. Certificates are suggested as a technical solution to establish trust, but they have their flaws. The certificate structure itself introduces vulnerabilities.

TLS web server authentication relies on the web public key infrastructure (PKI) to operate effectively. In the Web PKI, clients rely on a specific set of trust anchors, also known as a root store, to verify the identities of websites, along with their corresponding cryptographic public keys. However, the design of the existing Web PKI introduces a critical vulnerability where each trust anchor represents a single point of failure. This becomes a concern when malicious actors infiltrate hierarchical systems. Removing them becomes challenging, especially as they ascend the hierarchy, gaining more influence and authority.

If any of these trust anchors are compromised, it opens the door for malicious impersonation of a large number of web identities. Consequently, the security of the PKI system heavily depends on carefully selecting trust anchors for the root store.

This article will describe how to understand which trust anchors and certificates are trusted by default in the Red Hat Enterprise Linux (RHEL) root trust store, where RHEL derives these from and why deviations matter, and, how to remove a compromised certificate authority (CA) from the RHEL root trust store to avoid a potential security exposure.

The TrustCor Case: Unmasking the Threat of Compromised Certificate Authorities

An investigative article published by the Washington Post¹ highlights an alarming case involving TrustCor, an offshore company with ties to contractors working for U.S. intelligence agencies and law enforcement. The company has been linked to activities involving surveillance, data inception, and even connections to spyware.

TrustCor, a root certificate authority relied upon by major web browsers and tech companies to verify the legitimacy of websites, has come under scrutiny due to its affiliations with a spyware manufacturer. This manufacturer is known for selling communication interception services to U.S. government agencies, raising questions about TrustCor’s involvement in surveillance and data interception activities.

Upon further examination, doubts have arisen regarding TrustCor’s claims about its end-to-end encrypted email services. Experts have uncovered evidence that challenges these encryption assertions, casting doubts on the service’s security.

In a related incident, researchers discovered links between TrustCor and Panamanian company Measurement Systems. This company had been paying developers to include code into various seemingly harmless apps. This code recorded and transmitted users’ phone numbers, email addresses, and exact locations. It is estimated that these apps were downloaded more than 60 million times.

This story underscores the complexities and potential risks associated with root certificate authorities within the internet’s infrastructure. Certificate authorities play a critical role in ensuring the security and trustworthiness of online communications. As the article aptly puts it, “You can’t bootstrap trust, it has to come from somewhere […] Root certificate authorities are the kernel of trust from which it is all built on. And it will always be shaky, because it will always involve humans, committees and decision-making.”

In conclusion, the TrustCor case serves as a stark reminder of the intricate web of trust behind online security. The story highlights the critical role played by root certificate authorities in safeguarding our digital interactions. It underscores the need for constant vigilance and transparency. As we navigate this complex terrain, the ultimate goal remains preserving trust in an online world where trust is both the cornerstone and the challenge.

Zero Trust and Least Privilege: Securely Navigating the Intricate Web of Certificates

In the ever-changing landscape of digital security, two essential security concepts stand constant: zero trust and least privilege. These concepts are instrumental in enhancing security and mitigating risks in the digital landscape. However, adopting these principles requires proactive action, making it a challenging endeavor.

Let’s first explore the concept of zero trust. This approach challenges the traditional notion of perimeter-based security and operates on the principle of “never trust, always verify”. It rejects inherent trust and demands continuous authentication, authorization, and monitoring for every user, device, and network resource to ensure their legitimacy. By implementing zero trust principles, organizations minimize the potential damage caused by compromised trust anchors. However, it should be noted that operating system vendors may not naturally align with zero trust, necessitating organizations to continuously check for new CAs added with each update.

Next, let us delve into the concept of least privilege. This security principle revolves around granting only the minimum access rights and permissions necessary for users, devices, or applications to perform their intended tasks. Put differently, no one should have more access or capabilities than what is strictly required to get their job done. When a user or program with limited privileges is compromised, the damage they can cause is restricted to their designated tasks, preventing them from inflicting widespread harm. In the context of this article, applying the principle of least privilege entails limiting the number of certificates stored in the trust store to only those necessary for the system to function correctly, reducing the overall attack surface and potential vulnerabilities. However, software vendors may not offer the choice to adopt the least privilege principle by default, necessitating organizations to actively assess and manage the trust anchors with each software update or fix across all of the software deployed in the estate.

In summary, both zero trust and least privilege provide vital frameworks for establishing a robust security posture. They address trust vulnerabilities and ensure that access privileges are granted judiciously based on verified needs and the principle of granting the least possible privilege. However, organizations must proactively monitor for new CAs added in each update, as operating system vendors may not inherently align with these principles. By taking decisive action and incorporating these principles into their security practices, organizations can increase their resilience against potential security threats.

To this end, it is important to learn about root store families – the origins of the root stores that hold the certificate authorities. Understanding these root store families offers valuable insights into the broader context within which organizations navigate trust and security, and the intricacies of root store management.

Root store families: Exploring the Intricate Web of Certificates

The ecosystem of TLS root stores can be described as an inverted pyramid², with a majority of clients placing their trust in a handful of root programs, or root store programs. Currently, these are Apple, Google, Microsoft, and Mozilla/ Network Security Services (NSS), plus Oracle (Java) with a smaller outreach:

The research also points out that each root program plays a critical role in establishing trust for user agents by selecting the CAs included in their root store. It also highlighted some challenges associated with this approach, which are expected to intensify with the increasing adoption of TLS in various devices and applications:

Custom trust decisions While the CA/Browser Forum’s Baseline Requirements provide standards for CA behavior, such as secure identity verification procedures and operational security practices, the decision of which CAs to trust is left to the discretion of the root store operators. Each root store operator maintains custom CA inclusion and removal policies, as well as auditing processes to ensure compliance. These policies can vary in their level of stringency, with some operators implementing much more rigorous standards compared to others – and some trust decisions appear to be political.

Not fit-for-purpose usage X.509 certificates are used to represent the digital identity of a Certification Authority in root stores. These certificates not only contain identity information but also specify functionality constraints, such as validity periods that limit the duration of the root CA’s utility and Key Usage (KU) and Extended Key Usage (EKU) extensions that define the permitted roles (certificate signature, key agreement etc.) and trust purposes of the certificate (TLS client or server authentication, encompassing email signatures [S/MIME], code signing, and more). It is important to note that the Baseline Requirements only apply to “publicly trusted” TLS server certificates, and the requirements and procedures for other trust purposes may differ. Therefore, a root CA trusted for one type of identity verification may not necessarily be suitable or trusted for other purposes. Furthermore, certain root store operators like Mozilla and Microsoft, as well as end entities such as browsers, may impose additional restrictions alongside those defined by the CAs. Root programs that lack externalized trust policies have limited visibility into what their roots are trusted for. As a result, multi-purpose root stores can result in the unintended trust in root certificates for various purposes such as TLS server authentication or email encryption.

Update issues There are legitimate concerns regarding the update and customization of derivative root stores, which can lead to heightened security risks.

The TrustCor case serves as a prime example. This discovery led to the decision to distrust TrustCor’s CA certificates. While this change has been implemented by specific organizations (such as Mozilla by marking the certificates in question with a distrust date), the impact on Linux systems may be delayed due to the update process. Linux distributions typically follow a “bag of certificates” model, where all certificates in the root store are either trusted or not trusted. Nuanced trust decisions, such as trusting some certificates but not others, are not widely supported. Incorporating additional trust information from the root store into Linux distributions’ packages of root certificates is possible in theory, but practical challenges arise due to the lack of standardized formats and code to handle this information effectively. Consequently, the current practice does not involve capturing and including such additional information in the CA certificate packages³.

Exploring the RHEL root trust store: A Gateway to Secure Authentication?

In this article, the RHEL trust store serves as an example due to the widespread adoption of RHEL. Over 90% of Fortune 500 companies rely on RHEL, and it is deployed on nine million physical servers, comprising 16% of the global server installations⁴. A trust store contains signer certificates (also known as certificate authority certificates) that are used to verify the certificates received during TLS handshaking.

Like most Linux distributions, RHEL does not have its own CA certificate program. Operating such a program involves a significant legal framework as well as reviewing audits of various vendors. Instead, RHEL depends on the Mozilla program for SSL and email certificates and the Microsoft program for code signing certificates. In general, RHEL tries not to deviate from these. However, “RHEL packaged” certificate bundles may occasionally lag behind these.

The consolidated system-wide trust store in RHEL is located in the /etc/pki/ca-trust/ and /usr/share/pki/ca-trust-source/ directories. It is important to note that the trust settings in /usr/share/pki/ca-trust-source/ are processed with lower priority than settings in the /etc/pki/ca-trust/ directory. You can find more information about this in the Red Hat Enterprise Linux documentation⁵ on securing networks.

To obtain a list of all system trust anchors and certificates, use the trust list command:

RHEL 8.7

# trust list

To extract the certificates, you can use the following OpenSSL commands:
# openssl crl2pkcs7 -nocrl -certfile /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem | openssl pkcs7 -print_certs -noout

The RHEL trust store currently recognizes over 100 root certificate authorities, including three from TrustCor:

# Issuer: CN=TrustCor RootCert CA-1,OU=TrustCor Certificate Authority,O=TrustCor Systems S. de R.L.,L=Panama City,ST=Panama,C=PA # Serial Number:00:da:9b:ec:71:f3:03:b0:19 Trusted for SSL and Email from Mozilla.

# Issuer: CN=TrustCor RootCert CA-2,OU=TrustCor Certificate Authority,O=TrustCor Systems S. de R.L.,L=Panama City,ST=Panama,C=PA # Serial Number:25:a1:df:ca:33:cb:59:02 Trusted for SSL and Email from Mozilla.

# Issuer: CN=TrustCor ECA-1,OU=TrustCor Certificate Authority,O=TrustCor Systems S. de R.L.,L=Panama City,ST=Panama,C=PA # Serial Number:00:84:82:2c:5f:1c:62:d0:40 Trusted for SSL and Email from Mozilla.

The Mozilla root family, where RHEL derives its SSL certificates from, opted to set TrustCor certificates to DISTRUST_AFTER Nov 30, 2022. Which means Mozilla will not trust certificates issued from that CA after Nov 30, 2022. The CA-certificate will be removed when the last existing certificate has expired (probably 2025).⁶

This setting is mirrored in RHEL’s ca-certificates package, but only honored by NSS applications (including Firefox and Thunderbird) and won’t be honored by non-NSS clients such as wget or curl. All places will only be updated when RHEL picks up the change in their normal refresh.

In conclusion, using the default trust stores as-is is not secure as sanitization of compromised certificate authorities is performed neither promptly nor automatically. Users must take an active role in reviewing and managing their trust stores to safeguard their systems against potential threats.

The Journey to Secure Web Entities: How to Remove Compromised CA Certificates

There are three ways you can deal with a compromised CA certificate from the RHEL root trust store:

Option 1: Manually remove compromised CAs

To remove a CA from the RHEL trust store, you can use the trust anchor --remove command:

1. Run the trust list command to identify the pkcs11:id

# trust list

2. Remove the CA using the trust anchor command

# trust anchor --remove pkcs11:id=%53%ca%17%59%fc%6b%c0%03%21%2f%1a%ae%e4%aa%a8%1c%82%56%da%75

If you wish to remove all TrustCor certificates at the same time from your live trust bundle you can run the following:

# perl -e 'while(<>){if($_=~m/TrustCor/i){print$_;while(<>){last if length($_)==1;print$_}}}' </etc/pki/tls/certs/ca-bundle.crt >/etc/pki/ca-trust/source/blacklist/trustcor.pem

3. Update the root trust store

# update-ca-trust

For reference, please see How to install a CA certificate on Red Hat Enterprise Linux 7 and later? and the RHEL Security hardening guide.

Option 2: Adopt a zero trust policy

Revoke all publicly trusted CAs and selectively add back only the ones necessary for your specific use cases. This approach reduces the attack surface and ensures that only trusted CAs are relied upon as suggested by NIST SP 1800-16⁷:

CA trust

5.1.20 CA Trust by Relying Parties

[…] if certain systems acting as TLS clients are used only for internal operations, then they should trust only the certificate(s) from the internal CA(s). Furthermore, if certain TLS clients communicate with TLS servers from select partners, then certificates from only the CAs expected to be used by those partners should be trusted. Organizations should maintain an inventory of CA certificates trusted on all their systems, ensure only needed CAs are trusted, and maintain the ability to rapidly remove or replace CA certificates that should no longer be trusted.

Option 3: Wait for the vendor

Wait for Red Hat to pick up the change and fix the issue. To make Red Hat aware of this issue, I raised a case with Red Hat which resulted in Bug 2167809⁸.

Summary

The RHEL root trust store plays a crucial role in ensuring secure TLS web server authentication. However, the existing Web PKI design introduces vulnerabilities that can be exploited through compromised trust anchors. Understanding the trust settings in the RHEL root trust store and where RHEL derives its CA certificates from is essential for maintaining a secure environment. Recent events, such as the TrustCor case, highlight the importance of swiftly removing compromised certificate authorities. While RHEL strives to align with their trusted root programs, it is ultimately up to you, the user, to actively review and manage their trust anchors and take necessary actions to protect your systems from potential threats. Until software is configured out of the box with zero trust, or at least the option of deploying with zero trust, requiring you to explicitly configure least privileges, this problem will continue.

Special Thanks

I would like to thank Chris Schneider, Jackson Leonard, Kevin Grigorenko, Matthias Biniok, Steffen Lützenkirchen and Witold Szczeponik for their invaluable input and feedback.