As we found out on November 18, when a massive number of websites all rely on a common single point of failure, something like the Cloudflare outage translates to thousands of websites, apps, and services going offline. If you use Cloudflare for your app or website, you went down for several hours just like ChatGPT, X, Canva, and numerous online retailers.
If you’d found this page before Cloudflare recovered their service, it’s entirely possible your website would have been back online before many Cloudflare customers. Cloudflare is typically extremely reliable, so this isn’t a knock on their service. Think of this as a lesson in how to create a more resilient hosting strategy.
What caused the Cloudflare outage?
In the early morning hours US time, Cloudflare confirmed they were experiencing an issue impacting their entire network. This resulted in:
- DNS lookups failing
- Websites failing to load
- 500 errors
- Gateway errors
- Slow or stalled page loads
- No access to the Cloudflare dashboard
Cloudflare offers a comprehensive set of services, including DNS, CDN, routing, and security. Using any one of these services would have caused your website or app to fail today, even though your website hosting provider wasn’t having any issues.
After several hours of downtime, Cloudflare revealed on their blog that the issue “was triggered by a change to one of our database systems’ permissions which caused the database to output multiple entries into a “feature file” used by our Bot Management system. That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.”
Cloudflare’s crash became your crash too.
How can you prevent an outage caused by Cloudflare in the future?
You can skip to step 3 if you are reading this while Cloudflare is functioning correctly. The first two steps are intended to help you recover quickly if Cloudflare is currently experiencing an outage.
1. Check Cloudflare Status
Start by verifying whether Cloudflare is actually the issue. You can use the Cloudflare status page to check on their system health.
2. Update your DNS to direct to origin
This step assumes you registered your domain name somewhere other than Cloudflare’s domain registrar. You need to know the IP address of the server where your website is hosted, then follow these steps:
- Login to your Domain registrar and go to the settings where DNS is managed
- Change the A record for your domain to point to your website host and make sure you set the TTL to one of the shortest time options
- Turn off Cloudflare proxy in the Cloudflare dashboard (if you can access it)
3. Configure a multi-DNS failover
The long term fix to avoid a Cloudflare outage (or AWS or any other service) is to configure multiple DNS servers so that you can failover in the event one goes down. If you use Cloudflare services, this configuration could look like:
- Cloudflare as your primary DNS
- AWS Route53 as a secondary DNS provider
- DNSMadeEasy as a 3rd DNS provider option
- Automatic failover so that anytime Cloudflare has an issue, one of the other DNS providers will continue to provide support.
Another important consideration is making sure you know what the origin IP address is for your primary website. Store this somewhere so that you can easily access it when any of the services you rely on goes down.
Takeaways from the recent Cloudflare outage
Not unlike the recent AWS outage, the November 18 Cloudflare outage showed that big portions of the internet rely on single points of failure. Businesses of every size need a plan to be more resilient.
