30 September 2021

Let's Encrypt DST Root CA X3 expired. What to do for old devices and websites?

An expired root certificate could cause web access problems for old devices

letsencrypt CA X3 expire expired banner

It's not the electronic device apocalypse but some products may display error messages from Thursday. In question: the expiration of a Let's Encrypt digital security certificate scheduled for September 30, 2021 at 16:01 pm.

 

This file, called “IdentTrust DST Root CA X3”, is a root certificate of Let's Encrypt CA, a non-profit organization that has been helping democratize Internet encryption for free for several years. It's about to come out after twenty years of securing connections between devices and websites.

The consequence is that at the end of the month, some devices will have difficulty accessing the Internet. You may see error messages indicating that the connection is not secure. However, users will not be private of the Internet. As well as browsing with an obsolete browser or on an unsafe page they should be able to force the connection by choosing not to trust the certificate, using the Firefox browser, which issues its own certificates, or by updating their system.

ATTENTION: we have received several messages from “non-experts” users who have asked us what this message means “in a nutshell”. Realizing that there are many non-technicals, we can briefly summarize that if you use a browser that is too old on an operating system or a device that is too old you do not have the updated certification authorities (try to use Firefox on both mobile and desktop to solve if you can't update your device).

If you are website owner it means that unfortunately a slice of users will not be able to browse your site and possibly lose earnings / sales / contacts and in any case more or less important opportunities. This eventuality depends a lot on the target of your customers. If you are targeting a young audience, they will surely have the latest trendy smartphone and you won't have to worry too much about this aspect.

If, on the other hand, you are targeting older people, not too experienced housewives or professionals who may fall into the less "technological" range, you could lose up to 10% of traffic with obviously loss of earnings.

If you want to solve the problem directly without going into terminologies aimed at a technical audience / developers / webmasters / systems engineers, we invite you to contact us on this page.

However, this situation should not affect a large number of people.

As emphasized Scott helmets, the security researcher who highlighted this event on his blog, only older devices still in circulation are threatened. At Apple, they are targeted iPhones that haven't been updated for five years and have not yet been updated to iOS 10. All models launched from iPhone 5 can migrate to this version to correct the problem.

Users of older devices are likely to experience problems on the web starting today, September 30, at 16:01 pm. The expiration of a root certificate will in fact prevent almost all browsers from loading websites on these terminals. Older iPhones and Macs are potentially affected.

First of all, you should know that for the vast majority of Internet users, this September 30th will be a Thursday like any other. Those who own a Mac with MacOS 10.12.1 (Sierra) and later, as well as those who have an iPhone beyond iOS 10 (iPhone 5 can accommodate this version) will be spared, as well as Android 7.1.11 and later users, as well as those running Windows XP SP3 and later.

This means that users of products with older versions of their operating systems will have problems on the internet from tomorrow. For Macs, you can always change the certificate "by hand", from the Keychain Access application. It is necessary to replace theIdentTrust DST Root CA X3 with ISRG Root X1, provided by Let's Encrypt, whose effective date is until 2035. Further technical information is available on the OpenSSL website.

Another solution: use Firefox which comes with its own list of root certificates, remember Let's Encrypt. For all other devices, there will be no problems if they are updated regularly.

 

There is a long background

Seems like a shameless grip, but if you want really learn more about how certification authorities (CAs) and certificate chains work, you should consider joining me in the training practical TLS and PKI which I offer and which was created by Ivan Ristic , the creator of SSL Labs and author of Bulletproof SSL and TLS . For everyone else, this blog post and the details I'll link to should be enough to understand what's going to happen and why.

Ultimately, all certificates that power HTTPS on the web are issued by a CA, a trusted organization recognized by your device / operating system. Here you can see the list of "Trusted Root Certification Authorities" on my current Windows 10 device:

These certificates are built into your operating system and are usually updated as part of the normal update process for your operating system. The certificate here that will cause a problem is this, IdenTrust DST Root CA X3.

As you can see, time is running out and we are approaching the expiration date of September 30, 2021 but it's not just an expiration date, it's an expiration timestamp we call notAfter:

This is converted to BST for me, but if I parse the certificate using OpenSSL X509 you can see the UTC timestamp for expiration:

This gives us a fairly specific time for this certificate to expire:

Validity
            Not Before: Sep 30 21:12:19 2000 GMT
            Not After : Sep 30 14:01:15 2021 GMT

Once this root CA has expired, clients, such as web browsers, will no longer trust certificates issued by this CA.

In principle, the problem concerns devices that have not been updated for five years

Top Android, the affected devices are those that do not have the version 7.1 system, also launched five years ago. But a signed agreement between Google and a certificate authority will allow some even older devices to use the certificate. Devices launched ten years ago should continue to function normally, according to Let's Encrypt. As the smartphone's renewal cycle fluctuates between two and five years, depending on the market, the vast majority of users won't see any changes on Thursday.

In addition to smartphones, certificate expiration could cause problems for people who are still playing versions of the PS4 which have not been updated since version 5 of their firmware, which was launched in late 2017. Users of Mac which have not been updated since macOS 10.12.1 and PC even with Windows XP Service Pack 3 are affected.

Ultimately, we find here mainly machines that have not been updated for at least five years, which greatly limits the scope of the event.

Let's Encrypt has become popular in the meantime

In the last year alone, Let's Encrypt has significantly increased its market share and as a CA gets bigger, its certificates allow more of the web to operate and as a result, when something like this happens, they have the potential to cause more problems. This has nothing to do with what Let's Encrypt did or didn't do, but it still comes with the same underlying issue that devices in the ecosystem aren't updating as they should.

 

 

Given the relative size difference between Let's Encrypt and AddTrust, I have a feeling that IdenTrust's root expiration has the potential to cause more problems. Nobody really knows how much of a problem it could be, it could have similar consequences when AddTrust expired, or there could be some unforeseen circumstances and it could be a lot worse, your guess is as good as mine.

What is Let's Encrypt doing about it?

As I said above, this problem is not occurring due to something that Let's Encrypt has or has not done, it is happening because all certificates eventually expire and if the devices are not updated, they will not receive the new replacement certificates. That said, Let's Encrypt didn't sit and twist its thumbs as the expiration date approached, they worked hard trying to find a solution.

In April 2019 Let's Encrypt proposed switching to ISRG root, where Let's Encrypt had planned to move from the IdenTrust root to its own root, ISRG Root X1, which expires on June 4, 2035, giving us quite a number of years. The problem was that not many devices had received the necessary updates which include this new ISRG Root X1, released 4 years earlier in 2015! If a large selection of devices haven't received an update to include this new root certificate, they simply won't trust it. This is basically the same problem we are having now with the IdenTrust root expiration, because the client devices have not been updated, and have not even received the new ISRG Root X1. The transition was postponed.

A September 2020 Let's Encrypt had once again postponed the transition. They cited the following concerns:

Due to concerns about insufficient ISRG root propagation on Android devices, we have decided to move the date we will start serving a chain to our root to January 11, 2021.

 

This loosely translates to Android devices that haven't received an update for over 4 years, meaning those devices hadn't received ISRG Root X1 yet, meaning they didn't trust it. Let's Encrypt cannot pass the issue from the new root, but the IdenTrust root is still 1 year old and time is really short.

Eventually, something a little unexpected happened that could only reduce the serious impact of this event and make it a little more palatable. Since older Android devices don't check the expiration date of a root certificate when using it, Let's Encrypt may be able to continue chaining the expired root certificate without any problem on those older devices. This introduces some complexity in the future, but ultimately the goal is extend the compatibility of Android devices for Let's Encrypt certificates .

For this to work, Let's Encrypt had to get a cross-sign for their ISRG Root X1 certificate from the expiring IdenTrust DST Root CA X3, but that wouldn't have been of any help unless the cross-signed root was valid longer of the signature root, that is. The new ISRG Root X1 certificate is valid longer than the IdenTrust DST Root CA X3 that signed it!

This document page of Let's Encrypt contains a list of clients that trust only the IdenTrust DST Root CA X3 certificate and next there is the list of platforms that trust ISRG Root X1. I have merged these two lists to produce the following list of clients that will fail after IdenTrust DST Root CA X3 expires.

  • OpenSSL <= 1.0.2
  • Windows <XP SP3
  • macOS <10.12.1
  • iOS <10 (iPhone 5 is the lowest model that can go to iOS 10)
  • Android <7.1.1 (but> = 2.3.6 will work if ISRG Root X1 cross-sign served)
  • Mozilla Firefox <50
  • Ubuntu <16.04
  • Debian <8
  • Java 8 <8u141
  • Java 7 <7u151
  • NSS <3,26
  • Amazon FireOS (silk browser)

The platforms that I am still not sure about and that will need further investigation to see if they will fail after IdenTrust DST Root CA X3 expires are:

  • Cyanogen> v10
  • Jolla Sailfish operating system> v1.1.2.16
  • Kindle> v3.4.1
  • Blackberry> = 10.3.3
  • PS4 game console with firmware> = 5.00
  • IIS

The answer to the question "What will happen when the IdenTrust root expires?" depends on how widespread the types of customers listed above are. I don't know what's circulating out there on the web, and I don't even know what's hinging on these things. One thing I do know, though, is that at least something, somewhere, will break.

What problems for Hosting Providers deriving from the expiration of the DST Root CA X3 Let's Encrypt?

Surely the problem has not gone unnoticed considering the dozens and dozens of phone calls that the online Ticketing system and telephone support have already stormed us from 16 September 30th.

Some users with outdated devices such as some Mac OS Yosemite and some Chrome they used immediately complained of the error of the expired certificate that occurred when trying to reach the sites that adopted the free Let's Encrypt certificates.

We had difficulty in immediately focusing on the problem and being able to promptly argue both the reason for this and the relative solution.

However, that wasn't the only problem.

The apparently most difficult one concerned in fact some websites that adopted SMTP clients and that used the SSL or TLS protocol for forwarding outgoing mail to mail servers that used Let's Encrypt SSL certificates.

Let's imagine those classic web forms which, once they have been filled in and pressed the "Send" button, endeavor to send an email message to an address. Normally they use the PHP PHPMailer library, which, encountering an expired SSL certificate, refused the connection, avoiding the forwarding of the outgoing mail to the configured mail server.

However, in the most modern systems such as RedHat RHEL 7 and later that we use in the company, it was enough to update the Certification Authority certificates by launching the command:

yum update ca-certificates

It made it possible to update the list of Certification Authorities that logically excluded the expired Let's Encrypt CA X3 to the latest release.

For the most obsolete systems now in EOL for some time (End Of Life) we had instead to blacklist the expired CA to avoid negotiating with a certificate that is no longer useful by now.

What solutions for MacOS owners for certificate error?

net error MacOS

The only solution for older MacOS versions is to update the operating system to at least the El Captain release.

Alternatively, you can install the Firefox browser which, as we have already said, uses the CAs stored in the browser installation and NOT those of the operating system.

 

Which solution for Hosting Providers and websites?

If you have a website and want to keep compatibility with these old devices, the best advice we can give you is to upgrade to a certificate Commercial SSL DV Domain validated such as Comodo, Verisign, RAPIDSSL.

For example, we at Managedserver.it as providers of Hosting providers and mail services with their secure and encrypted protocols including:

SMTP AUTH: Port 25 or 587
SMTP SSL: Port 465
SMTP StartTLS: Port 587
POP3: Port 110
IMAP: Port 143
IMAPSSL: Port 993
IMAP StartTLS: Port 143

we decided to rely on a fast and economical as well as robust and compatible Positive SSL from Comodo bought for two years at just about $ 5 a year on SSLS.COM

The cost itself is really low if you work on a single domain, however the installation procedure on Webserver and especially SMTP Server and IMAP and POP3 Server may not be within everyone's reach, especially if a beginner perhaps with a control panel. amateur like Plesk or cPanel which we have widely talked about (badly) if adopted in professional environments.

However, we are talking about a commercial certificate which, although extremely cheap, may not be the ideal solution (or simply the one desired), which requires a certain amount of time, money and energy to:

  1. Buy the SSL certificate online and pay for it.
  2. Generate the CSR in order to subsequently generate the certificate (normally through the openssl utility)
  3. Verify domain ownership via DNS or by receiving mail to addresses such as admin@domainname.it
  4. Wait for the validation and the e-mail with the certificate
  5. Install and configure it.

In short, a series of operations that at best require at least 30 minutes per single domain. If you want to operate on large volumes and many domains as in the case of customers who have many domains, the fastest solution is to use CloudFlare certificates as we will see below.

Use CloudFlare SSL certificates in reverse proxy mode.

If you don't want to spend a single euro but you know how, you can enable CloudFlare in Reverse Proxy mode and their chain of certificates that it is not based on LetsEncrypt.

CloudFlare caches static site content and distributes it across a network of hundreds of servers around the world, on a global CDN powered by 30 data centers. When the user visits the website, he doesn't have to reach the origin server, because the cached content is delivered to him by the Cloudflare server closest to him, loading twice as fast! It's one of the easiest ways to improve performance and reduce the load on web servers.

Nothing happens at the file and system level of your site, all data that passes through the DNS system is intercepted and managed by the CloudFlare network which in this way applies some important optimization operations on Javascript, images, content for the mobile, for browsers.

  • Static page cache
  • CDN
  • Image optimization
  • Javascript optimization
  • Mobile optimization
  • Balanced content

Obviously, have the foresight to connect the APIs with the related modules for the main CMS.

If you don't know how to do it, please contact us for a system consultancy. We will solve the problem in a few minutes.

 

Do you have doubts? Don't know where to start? Contact us!

We have all the answers to your questions to help you make the right choice.

Chat with us

Chat directly with our presales support.

0256569681

Contact us by phone during office hours 9:30 - 19:30

Contact us online

Open a request directly in the contact area.

INFORMATION

Managed Server Srl is a leading Italian player in providing advanced GNU/Linux system solutions oriented towards high performance. With a low-cost and predictable subscription model, we ensure that our customers have access to advanced technologies in hosting, dedicated servers and cloud services. In addition to this, we offer systems consultancy on Linux systems and specialized maintenance in DBMS, IT Security, Cloud and much more. We stand out for our expertise in hosting leading Open Source CMS such as WordPress, WooCommerce, Drupal, Prestashop, Joomla, OpenCart and Magento, supported by a high-level support and consultancy service suitable for Public Administration, SMEs and any size.

Red Hat, Inc. owns the rights to Red Hat®, RHEL®, RedHat Linux®, and CentOS®; AlmaLinux™ is a trademark of AlmaLinux OS Foundation; Rocky Linux® is a registered trademark of the Rocky Linux Foundation; SUSE® is a registered trademark of SUSE LLC; Canonical Ltd. owns the rights to Ubuntu®; Software in the Public Interest, Inc. holds the rights to Debian®; Linus Torvalds owns the rights to Linux®; FreeBSD® is a registered trademark of The FreeBSD Foundation; NetBSD® is a registered trademark of The NetBSD Foundation; OpenBSD® is a registered trademark of Theo de Raadt. Oracle Corporation owns the rights to Oracle®, MySQL®, and MyRocks®; Percona® is a registered trademark of Percona LLC; MariaDB® is a registered trademark of MariaDB Corporation Ab; REDIS® is a registered trademark of Redis Labs Ltd. F5 Networks, Inc. owns the rights to NGINX® and NGINX Plus®; Varnish® is a registered trademark of Varnish Software AB. Adobe Inc. holds the rights to Magento®; PrestaShop® is a registered trademark of PrestaShop SA; OpenCart® is a registered trademark of OpenCart Limited. Automattic Inc. owns the rights to WordPress®, WooCommerce®, and JetPack®; Open Source Matters, Inc. owns the rights to Joomla®; Dries Buytaert holds the rights to Drupal®. Amazon Web Services, Inc. holds the rights to AWS®; Google LLC holds the rights to Google Cloud™ and Chrome™; Facebook, Inc. owns the rights to Facebook®; Microsoft Corporation holds the rights to Microsoft®, Azure®, and Internet Explorer®; Mozilla Foundation owns the rights to Firefox®. Apache® is a registered trademark of The Apache Software Foundation; PHP® is a registered trademark of the PHP Group. CloudFlare® is a registered trademark of Cloudflare, Inc.; NETSCOUT® is a registered trademark of NETSCOUT Systems Inc.; ElasticSearch®, LogStash®, and Kibana® are registered trademarks of Elastic NV This site is not affiliated, sponsored, or otherwise associated with any of the entities mentioned above and does not represent any of these entities in any way. All rights to the brands and product names mentioned are the property of their respective copyright holders. Any other trademarks mentioned belong to their registrants. MANAGED SERVER® is a registered trademark at European level by MANAGED SERVER SRL Via Enzo Ferrari, 9 62012 Civitanova Marche (MC) Italy.

Back to top