Table of contents of the article:
The subject of this article was born to perfection, after several weeks of "lack of inspiration" on topics related to web performance. Either because Black Friday absorbed the energies for customer e-commerce scaling, or because we are putting a lot of meat on the fire with the development of plugins for Prestashop and Magento, as well as our dedicated CDN solution for adaptive images, but above all because we talked a lot about the technologies at work and in use in our specific software stack for WordPress.
Anyone who has read and followed the articles and posts published over time has certainly noticed how there are regular recurrences on technical articles when it comes to WordPress Hosting or Linux Hosting in a more general way.
We always end up talking about NGINX and Varnish as the ultimate combo when it comes to performance and speed, often forgetting how important painstaking tuning is to obtain adequate values such as the best possible TTFB.
We talked for a long time the importance of a TTFB as low as possible, and how important it is to get low latency; however, we have never been able to document with a differential analysis two identical sites with the same stack and a further optimization of the software stack and a specific tuning for NGINX Web server.
The specific case born by chance.
Last evening we posted a post on the private Facebook wall where TTFB was measured using the utility SpeedVitals.com, an online service that allows you to measure TTFB from multiple locations around the world.
From what was initially a "sterile" post, written on a gloomy Sunday and without too many goals, other than to externalize a small Vanity Metrics and "make us beautiful", several comments arose from there until late in the evening in which many they responded by posting their scores and the evaluations that this online tool returned.
Among the 26 comments that you can see at the bottom right of the image above, we were particularly struck by the comment of one of our customers who showed a particularly high TTFB when compared with the one posted in the screen above.
In particular, it was highlighted that the evaluation returned was of grade B and that the scanning time of the Google spiders was not exactly very low, reaching 141ms.
Considering that the underlying software stack was practically the same, it was certainly necessary to go and review what the specific problems of the configurations of the Varnish and NGINX services were to try to improve and optimize everything as best as possible.
Analysis of the metrics of the site in question.
By chance, the problem raised was under our total control and therefore, even if on Sunday, we wanted to go into a "file job" aware that it would have been a tempting and unrepeatable opportunity to show and demonstrate how a superfine tuning would have been able to bring tangible benefits and end up documenting everything with this post which is nothing more than a differential analysis between the pre and post tuning situation.
We have therefore decided to use a utility called ByteCheck (bytecheck.com) which allows us to measure the Time To First Byte and break down the total time into the various steps that make up the total TTFB.
Starting from the DNS resolution, to the connection time, to the establishment time of the encrypted connection, up to the waiting and delivery times of the contents, as we can see in the following image, in which it is evident in a rather evident and indisputable way that there is a particularly high waiting time of 240ms.
Objectively one wonders what the webserver (or the entire stack) does in those 240ms, too low a time to assume that the Varnish static cache is not working correctly, and at the same time, too high a value to assume that the software stack is functioning normally as best as possible.
We therefore wanted to test that the static cache was working correctly, using the Linux CURL utility, and examining the response from the HTTP headers.
[root@lawebstar ~]# curl -I https://www.ilmiocaneleggenda.it HTTP/1.1 200 OK Server: nginx Date: Mon, 28 Nov 2022 00:26:46 GMT Content-Type: text/html; charset=UTF-8 Connection: keep-alive Vary: Accept-Encoding Vary: Accept-Encoding Strict-Transport-Security: max-age=86400; includeSubDomains X-Content-Type-Options: nosniff X-Frame-Options: DENY X-XSS-Protection: 1; mode=block Links: ; rel=preload; as=script Links: ; rel=preload; as=script Links: ; rel=preload; as=script Links: ; rel=preload; as=script Links: ; rel=preload; as=script Links: ; rel="https://api.w.org/" Referrer-Policy: no-referrer-when-downgrade X-Cacheable: YES X-Varnish: 2.2.0 12 Age: 2 Via: 32853419 varnish X-Cache: HIT
The answer in question left no doubt that the Varnish Static Cache was working correctly, having cached the response for 4656 seconds and that the static cache returned a HIT and not a MISS and therefore found (and delivered) the already stored response in the cache.
The problem, therefore, must have been upstream, according to the "sandwich" logic in which NGINX is both the webserver on which the website runs, and the reverse proxy terminator of the HTTPS connection, considering that Varnish does not support HTTPS and needs of an HTTPS terminator according to the following scheme.
We have therefore completely revised the NGINX configuration both globally and for individual Virtual Hosts, and by meticulously adjusting parameters such as buffers, as well as enabling some well-documented features in the latest NGINX versions starting from 1.22 onwards, we have obtained a decidedly optimal value. bringing the waiting time from 247ms to 124ms, or effectively halving the final TTFB.
Obviously, being a hosting company, we apologize in advance if we do not show the specific configuration files and document the changes made in detail, being in fact know-how that we tend not to disclose for obvious business reasons.
The importance of low TTFB and the benefits you will get from this tuning.
We have already talked about the importance of TTFB as we initially mentioned in this article: What is TTFB? How to improve the Time To First Byte, however it is good to highlight how Google with its parameters Core Web Vitals, has decreed that a TTFB below 200ms can be considered good, and a higher TTFB that needs improvement.
Someone could object or ask what are the real advantages of a TTFB that is compliant with Google's expectations, and if it is really so important to have a TTFB below 200ms.
What is certain, and which you will need to pay close attention to, is that the scanning frequency of Google spiders and TTFB are directly related.
You can see it from a history like above of Search Console.
You can understand from the two blue and orange lines, that as the response time increases, (TTFB orange line), crawl requests from Google decrease (Blue line).
Conversely, as TTFB response time decreases, Google crawl requests increase.
If you're looking to know Google's crawl stats, in Search Console, go to "Settings" at the bottom of the left column. In the Scan section, you will find the “Open Report” button for the scan statistics. When the report is open, you can select the “Average Response Time (ms)” checkbox to add the metric to the graph.
Google uses 200ms as the cutoff value in other tests and reports, before they display a warning that the response time is high. Response time varies depending on your hosting, site quality and general configuration of cache, CDN, ed in this case the tuning of the webserver.
Conclusions and observations
From this real and well-documented case, we can observe how the use of the right tools may not be sufficient to obtain the best results, which can instead be obtained by using the right tools, but also and above all by using the right configurations.
It is also useful to understand how the right configurations of months or years ago can suddenly be deemed no longer adequate or even surpassed by new metrics and new requirements that require a continuous cycle of adaptations to new requirements.
We will keep this case study under observation to evaluate also in terms of indexing with the SeoZoom utility, how this optimization may have impacted positioning and indexing, evaluating the graph before and after November 28th, the date on which we worked to improve the response of the two sites in question on the same server (ilmiocaneleggenda.it and ilmiogattoeleggenda.it) and respectively evaluating the trend of the two sites in a future differential analysis.
