Table of contents of the article:
In the world of web hosting and infrastructure management, AWS (Amazon Web Services) is often considered the benchmark for scalability and reliability. However, like any powerful tool, the success of AWS depends on its proper configuration and management. This article analyzes a concrete case of an astronomy site, passionastronomia.it, which suffered a 4-hour downtime despite being hosted on AWS. We will explore the causes of the problem, the use of CloudFront and CloudFlare, and the importance of Full Page Cache to improve performance and reduce resource consumption.
The Case: passionastronomia.it
A few months ago, one of our valued customers with a monthly traffic of approx 50 million page views, reported to us an emerging astronomy site that periodically had downtime problems due to too much traffic.
He, who followed the Passione Astronomia page, was informed by the posts published on Facebook by Passione Astronomia itself. Every time a post caused an overload and the site went offline, they reported it by bragging that the server couldn't handle the high traffic. However, for the owner of a site used to managing 50 million page views per month, with peaks of over 3 million per day, it is quite "curious" to see how a site that manages about a tenth of them, or about 6 million per month (according to SimilarWeb estimates), can go offline so frequently without anyone working to solve the problem.
This is really a problem, since normally, when a downtime occurs and the traffic directed to the site is unable to contact it or accumulates numerous 500 errors, browsers that use Chromium (the engine behind Chrome), obviously including Chrome , report the incident to Google. Consequentially, Google will send less traffic to that site in the following hours, knowing that something is not working properly. This can have a significant impact on the visibility and performance of the site, further penalizing its traffic and reliability in the eyes of users and search engines, as well as obviously damaging its image.
Furthermore, even when the site is online, it does not seem to guarantee who knows what performance with a Time To First Byte of around 500ms at 8 in the morning with the site practically little visited and with no posts launched that generate traffic. Below you can, for example, view the TTFB test for Europe with the SpeedVitals.com TTFB test
Below you can view our TTFB of Managedserver.it, a site that we take care of obsessively to obtain the best performance.
We are talking about a TTFB (Time to First Byte) that is 10 times lower and well below the maximum 200 ms that Google considers acceptable before reporting that the server response time is too high, indicating the need to "Improve the server response time”.
Furthermore, the slowness of such a high TTFB, to the point of frequently becoming unattainable at certain times of the day, including prolonged downtime, negatively affects the Core Web Vitals which we can see from the Google PageSpeed Insight report below.
A high Time to First Byte (TTFB), as shown in the image with a value of 2,3 seconds, can negatively affect other parameters of the Core Web Vitals. A high TTFB means there is a significant delay before the browser starts receiving data from the server. This delay contributes to a high First Contentful Paint (FCP) (3,4 seconds) and a high Largest Contentful Paint (LCP) (4,7 seconds), because the initial rendering of the page is delayed. Additionally, a high TTFB can negatively impact Interaction to Next Paint (INP) and Cumulative Layout Shift (CLS), as slow page loading can lead to a less fluid user experience and more frequent layout shifts.
Accustomed to managing significant traffic peaks, even over 2 million visits per hour and 200 thousand per minute, and specialized in the optimization of Core Web Vitals, we have offered our services and our expertise to the owners of passionastronomia.it by contacting them to propose managed management of their infrastructure or migrate to our high-performance hosting, however, our offer was probably misunderstood and underestimated, as well as rejected without too many compliments and mincing words.
Yet our site boasts an optimal TTFB of just 22ms in Italy and a PageSpeed that brilliantly passes the test of Core Web Vitals as you can see below.
With a significant publishing client portfolio and the numbers mentioned above, which could be assessed impartially using the tools made available by Google, it seemed incredible that, in a moment of difficulty characterized by continuous downtime, they hadn't taken the opportunity to resolve all the problems in a flash. This could have happened at a cost that was probably half that of their current supplier, who often saw them offline or performing poorly, as indicated by the tests above.
Passioneastronomia.it and WordPress
Passioneastronomia.it is an editorial site developed in WordPress, based on PHP and MySQL. Although these technologies are now considered dated compared to more modern and asynchronous languages and technologies such as Node.js, Go, or NoSQL databases such as Cassandra, MongoDB or REDIS, they continue to be widely used. PHP, for example, was developed in the 90s and has undergone numerous updates over time, but remains less performant in high-load situations than modern asynchronous languages. Similarly, MySQL, despite being a robust relational database management system, can encounter difficulties in terms of scalability and performance when handling large volumes of data and concurrent requests.
Despite these limitations, WordPress remains the most versatile and easy-to-implement content publishing system available today. Its large developer community, myriad of available plugins, and intuitive interface make it a popular choice for small blogs and large businesses alike. Even major newspapers such as Il Fatto Quotidiano they use WordPress as their CMS for editorial publication, demonstrating that, with the right configuration and optimization, it is possible to obtain excellent performance even with technologies considered less modern.
However even NASA, the highest expression in the collective imagination of scientific dissemination, uses WordPress as a CMS.
Passioneastronomia.it initially (at least since they told us about it about a year ago), was hosted on SiteGround, a hosting service that is not always able to handle traffic of this level. Then in April, the site moved to AWS, the choice of the market and many big names such as Netflix, Airbnb, Spotify, Twitch, LinkedIn, Adobe, Slack and the BBC.
- Netflix: Uses AWS for video streaming, data storage and analytics.
- airbnb: Leverages AWS to manage its global hospitality platform and online marketplace.
- Spotify: Use AWS for streaming music delivery, data analytics, and machine learning.
- Twitch: The Amazon-owned video streaming platform relies on AWS for live video transmission and data management.
- LinkedIn: Although some of its infrastructure is internal, LinkedIn uses AWS for some of its data storage and analytics services.
- Adobe: Offers its Creative Cloud services and other SaaS applications through AWS.
- Slack: Uses AWS for its messaging and collaboration platform.
- with the BBC: The BBC uses AWS for on-demand video content delivery and data management.
Despite the power and scalability of AWS,passionastronomia.it continued to experience downtime, culminating in a 4-hour outage on May 16, from 19pm to 00pm, monitored by our Uptime Kuma tool.
The causes of Downtime
The main problem lies not in AWS as an infrastructure, but in the configuration and management of the service. AWS offers a wide range of services, including EC2, databases, load balancers, distributed file systems, and CDNs like CloudFront. However, if used inappropriately, AWS can lead to serious performance problems.
The passionastronomia.it site did not have an adequate Full Page Cache system such as CloudFront or the Pro or Business version of CloudFlare with HTML Cache support, or even a “simple” Varnish. The site's response headers clearly show that HTML caching was not enabled:
HTTP/2 200
date: Thu, 16 May 2024 22:02:18 GMT
content-type: text/html; charset=UTF-8
set-cookie: AWSALBTG=gkrvkTmzuBE7uhKteG6ihiCGQH60BIdF48ki+7cvKP9ia2ltc4cAgn5dVM5l+/WaO8fbb8dzylYF1OYP7PZnmhHdLsauuVLuLntiKviIt8EAxKNbM3yBSyKqrMaGu1SXAQaPGkLnjoHwqz3OkmDHAVqvBB0V3v4d0WOcshbhqixspvTJTic=; Expires=Thu, 23 May 2024 22:02:17 GMT; Path=/
set-cookie: AWSALBTGCORS=gkrvkTmzuBE7uhKteG6ihiCGQH60BIdF48ki+7cvKP9ia2ltc4cAgn5dVM5l+/WaO8fbb8dzylYF1OYP7PZnmhHdLsauuVLuLntiKviIt8EAxKNbM3yBSyKqrMaGu1SXAQaPGkLnjoHwqz3OkmDHAVqvBB0V3v4d0WOcshbhqixspvTJTic=; Expires=Thu, 23 May 2024 22:02:17 GMT; Path=/; SameSite=None; Secure
cf-edge-cache: cache,platform=wordpress
link: <https://www.passioneastronomia.it/wp-json/>; rel="https://api.w.org/"
link: <https://www.passioneastronomia.it/wp-json/wp/v2/pages/49>; rel="alternate"; type="application/json"
link: <https://www.passioneastronomia.it/>; rel=shortlink
vary: Accept-Encoding
cf-cache-status: DYNAMIC
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=dvOaq3CI0sDGSeXsjICpJUT0nF1zmp%2BFj1TKPfqJKvyRd%2FuybtNnkc9FKE6SLu7CldFY7brUo1HZwF1kvPoDyisecYzzZC3aI%2FlrjkKCKcs%2BD4LVRylVZMmSQViSYJRA6fO9JU1sg2ejKUk%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 884ea6b518c20e4d-MXP
alt-svc: h3=":443"; ma=86400 The Free version of CloudFlare does not support HTML Cache in DYNAMIC mode, causing excessive load on AWS servers with continuous HTML requests passing directly through PHP processes and database connections.
In the first case, when the load increases significantly, the origin Web Server reaches saturation, being unable to even respond and causing the reverse proxy of the CloudFlare CDN to time out as in the following image:
In the second case, even if the Web Server manages to respond for a few moments, certainly the load of SQL queries to the MySQL database will return a connection error to the Database as in the screenshot below.
More succinctly, it is correct to say without going too deeply into the world of DBMS, management of tables, queries and indexes that this lack of adequate caching has several negative implications, especially for high traffic sites such aspassionastronomia.it.
Implications of HTML Cache Lack
- Increased load on Origin Servers:Without HTML caching, each HTML page request must be processed directly by the origin servers. This means that each visit to the site activates PHP processes to generate the dynamic content of the page, significantly increasing the load on the servers. As a result, the server must handle a large number of concurrent requests, leading to rapid exhaustion of available resources.
- Database overload:Each HTML request can involve a series of database queries to retrieve the data needed to generate the page. As traffic increases, the number of database connections can exceed the manageable limit, causing the database to slow down or even crash. This database overload is often the root cause of prolonged downtime.
- High Response Times:Without caching, the time required to generate an HTML page can be significantly longer. Each request requires loading and executing PHP scripts, interacting with the database, and dynamically generating content. This process is much slower than serving a cached page, leading to high response times and a suboptimal user experience.
- Risk of Downtime:When origin servers are constantly under pressure due to the high number of requests, the risk of downtime increases. Servers can become unresponsive or even crash if the load exceeds their capacity to handle. This is exactly what happened to passionaastronomia.it, where the site experienced significant downtime due to the inability to handle high traffic without a proper caching system.
Inappropriate Use of AWS EC2
Although AWS EC2 is a powerful solution, using an EC2 instance, even a large Etra, without proper configuration can be ineffective. An EC2 instance is nothing more than a virtualized instance with a certain amount of cores and RAM, which offers significant computational resources. However, without proper traffic management and caching, these resources can quickly become overloaded. Poor configuration can turn a powerful instance into a bottleneck, unable to handle high traffic spikes.
For example, in the absence of load balancing mechanisms, a single server can become overwhelmed with too many concurrent requests, rapidly exhausting available CPU and memory. Furthermore, without an effective caching strategy, each user request must be processed completely by the origin server, involving PHP processes and database queries to generate the dynamic pages. This not only increases response times, but it also puts a strain on the database, which can become a critical failure point although as we can see, the Database is hosted on a Managed Database service such as RDS.
MySQL RDS (Relational Database Service) by Amazon AWS is a managed database service that makes it easy to set up, operate, and scale a MySQL database in the cloud. With MySQL RDS, Amazon takes care of administrative tasks such as hardware, operating system, database patches, and backups. This allows users to focus more on application development and less on infrastructure management.
Even the best EC2 instances can therefore fail if they are not supported by a solid backend architecture. To maximize the efficiency of an EC2 instance, it is essential to implement application-level caching systems and use CDNs like CloudFront or CloudFlare to distribute the load. Additionally, configuring Auto Scaling to automatically adapt to traffic changes can prevent sudden overloads. Only with careful management and optimal configuration, an EC2 instance can fully exploit its potential, ensuring high performance and reliability.
What are CloudFront and CloudFlare?
CloudFront
CloudFront is a CDN (Content Delivery Network) offered by AWS that delivers content around the world, reducing latency and improving page loading speed. This service is particularly useful for high-traffic sites such as passionastronomia.it, as it is capable of caching entire HTML pages, reducing the load on origin servers and significantly improving the user experience. Delivering content via a global network of edge nodes brings data closer to end users, resulting in faster response times and greater reliability. Additionally, CloudFront offers advanced features like DDoS protection and integration with AWS Shield and AWS WAF, which help improve site security. Using CloudFront also allows you to better manage traffic spikes, ensuring that origin server resources are not overloaded during periods of high demand. Its flexibility and scalability make it an ideal solution for improving site performance and offering a smoother and faster browsing experience to users.
CloudFlare
CloudFlare is another CDN service that offers a wide range of features to improve website performance and security. Unlike CloudFront, CloudFlare offers several caching options, including HTML Cache for Pro and Business versions.
The benefits of CloudFront and CloudFlare can be very similar, if not overlapping, CloudFlare's best-known advantage over CloudFront is that CloudFlare offers a Flat pricing plan with a basic plan of $25 per month which would have been more than enough to make the passionastronomia.it site perfectly compliant with the traffic peaks it receives on a daily basis.
CloudFront, on the other hand, offers a pay-as-you-go plan so the cost is directly proportional to the data moved.
In fact, both CloudFlare and CloudFront offer a Full Page Cache service in PaaS (Platform as a Service) mode, distinguishing themselves from Self Hosted solutions such as Varnish Cache. This difference is significant, as a PaaS service like the one provided by CloudFlare and CloudFront eliminates the need for you to manage and maintain caching infrastructure. With PaaS solutions, cache deployment and management are automated and integrated into the service, reducing administrative overhead and allowing teams to focus on other critical tasks.
For example, CloudFlare and CloudFront offer user-friendly interfaces for configuring caching rules, manage invalidations and monitor site performance, all without requiring in-depth technical server management skills. In contrast, Self Hosted solutions like Varnish Cache require manual configuration and ongoing management of the caching infrastructure. This entails the need to have qualified technical personnel capable of installing, configuring, monitoring and updating the caching software, as well as managing the physical or virtual servers on which it operates.
PaaS solutions from CloudFlare and CloudFront also offer a significant advantage in terms of scalability and reliability. Being cloud services, they can leverage the global network of distributed nodes to deliver high performance and low latency to end users, regardless of their geographic location. Delivering content across a global network of edge servers reduces latency and improves page load times, providing an optimal user experience.
Furthermore, CloudFlare and CloudFront integrate advanced security features, such as DDoS protection and integration with SSL certificates, which can be managed and configured directly from the platform without requiring manual intervention. This level of automation and integration makes it easier to secure web applications and provides greater peace of mind for site owners.
While Self Hosted solutions like Varnish Cache can offer more granular and customized control over cache configuration, they require a significant commitment in terms of resources and technical expertise. On the other hand, CloudFlare and CloudFront PaaS solutions provide a complete, automated and highly scalable service that simplifies cache management, reduces administrative burden and offers a higher level of performance and security.
Full Page Cache what is it?
Full Page Cache (FPC) is a caching technique that stores the entire HTML page generated by a website. This technique is essential for high-traffic sites as it drastically reduces the number of requests to the origin server, reducing the workload of PHP processes and database connections.
How it works
When a user visits a web page, the server generates the HTML page and caches it. Subsequent visits to the same page are served directly from the cache, avoiding page regeneration and reducing response times.
Conclusion
The lack of an HTML cache is a classic example of how the absence of an adequate caching strategy can put a strain on even the most robust infrastructures like AWS. For sites with high traffic, implementing an efficient caching system is crucial to reduce the load on origin servers, improve response times and prevent significant downtime. In this specific case, it would have been enough to use the $25 monthly version of CloudFlare, configured by a good systems engineer, to solve the downtime problems. Even better, implementing a Varnish Cache system with CloudFlare placed in front would have allowed for a double layer of cache. This configuration would have offered maximum performance at a cost that was certainly lower than the current one. Using such solutions, it is possible to achieve significant improvements in site performance and stability, ensuring an optimal user experience even during peak traffic.
This shows that beyond the provider of the services you decide to use, what really matters is the technical competence and expertise of the technicians who are responsible for implementing the best caching strategy in order to improve performance and content delivery.
Below is the traffic of our client's site that the editorial team of Passionastronomia.it reported to us for problems that we could have solved in 1 hour of work and consultancy, probably saving around 75% of the budget they are spending on AWS to go offline. But as they say in these cases, whoever is the cause of his evil should cry for himself.
It will be funny when the "bills" arrive from AWS and in particular from Amazon RDS for MySQL which is a Pay Per Use service and which will probably increase the costs significantly. Systems engineering is not child's play and as you can see inexperienced technical consultants can not only cause damage, but also drain your budget, not knowing how much more convenient it could be to spend by investing in CDN and Full Page Cache, rather than in Managed Database services. like AWS RDS.
We told him…
Updates two months later.
Following this post and the present ruminations, we believed it was correct (although not necessary) to share some technical suggestions with the Passione Astronomia company. We therefore sent an email that included extremely precise and useful technical advice for the current hosting scenario on Amazon AWS, therefore advising to install either a server-side Full Page Cache such as Amazon's CloudFront, or to enable the CloudFlare HTML Cache that they are already using in Free mode, perhaps using CloudFlare's APO directly at a low cost of around 5 dollars per month.
We confidently expected that within a few days we would see our precious technical suggestions implemented, however after almost two months, the analysis of the response headers shows no changes taking place, not even the shadow of Cloudfront, of CloudFlare which not even a shadow of HTML cache and a response time exceeding the maximum 200ms tolerated by Google.
The proof of the 9 was given by the downtime of 8 July 2024, that is, yesterday, the evening in which our Uptime Kuma monitoring system reported to us a prolonged downtime of the site in question, with a total and complete offline of 2 hours and 20 minutes , as we can see from the following dashboard.
We were quite shocked by the fact that our advice, completely disinterested and based on years of experience, was neither received nor implemented. It is particularly surprising considering that some of the largest and most important publishing companies on the Italian scene are willing to pay several thousand euros to solve problems of stability, speed and performance that we are able to deal with effectively.
This leads us to a legitimate reflection: are the operators in the sector, as well as our possible "colleagues", really aware of the cornerstones necessary for the management of an important and successful project like this? Perhaps the time has come to think seriously about establishing a professional register with a state exam to qualify for professions such as systems engineering. Such a step would guarantee high standards of competence and responsibility, ensuring that only the most trained professionals can operate in a sector so crucial to the success of modern businesses.
