Table of contents of the article:
We are excited to announce paint.js, the Open Source integration under the AGPL license, of support for Varnish in the Ghost CMS, which allows the Selective purge of Varnish cache when a new post is published or an existing one is edited. This improvement ensures that the content served to users is always up to date and reflects the latest changes in real time. To make this integration possible, we have developed in MANAGED SERVER SRL, a middleware called Paint.js, a server-side middleware written in Javascript for nodejs, designed to interface with Varnish Cache via Ghost CMS webhooks. This middleware listens for content change events in Ghost and sends purge requests to Varnish, ensuring the cache is always up to date.
This integration is particularly useful since Ghost does not natively offer support for a server-side Full Page Cache. Varnish, with the help of Vernice.js, can therefore be of great help for high-traffic blogs, both to support traffic spikes and to reduce response times such as TTFB (Time To First Byte).
Varnish Cache is an HTTP accelerator designed for dynamic, content-heavy websites. It works as a reverse proxy that caches web server responses, improving performance and reducing server load. Thanks to Vernice.js, we can take full advantage of Varnish's capabilities to deliver content that is always fresh and fast.
Vernice.js ensures that every time content is edited or published in Ghost, Varnish's cache is updated immediately, keeping the content served fresh. Using Varnish reduces load on the backend server and improves response times (TTFB), a crucial benefit for high-traffic blogs. This configuration helps manage traffic peaks without compromising site performance, making the site more reliable and faster.
By implementing Vernice.js, we can fully leverage the power of Varnish to improve the performance and reliability of our Ghost site. This ensures an optimal user experience even during periods of heavy traffic, keeping content always up to date and significantly reducing response times.
Unlike other solutions found online on the Web and on the various GitHub repositories, which tend to clean the entire Cache when a post is updated or a new post is published, our paint.js middleware is particularly scrupulous in carrying out a granular purge and selective of only updated content. For example, if we had a Ghost Blog populated by 100 thousand posts, and we updated a post by modifying its content, paint.js will take care of purging the home page and the URL of the updated post, leaving all the other posts in cache.
This allows you to obtain a very high HIT RATIO, making the cache particularly functional with all the resulting advantages.
What is Varnish?
Varnish is an HTTP accelerator designed for dynamic websites. It works as a reverse proxy that caches web server responses to improve performance and reduce server load. This means that once a web page has been requested and stored in the Varnish cache, further requests for that same page are served directly from the cache, without having to go through the backend server. This not only speeds up the delivery of content to users, but also significantly reduces the load on the server, allowing it to handle a much larger number of simultaneous requests.
Varnish is particularly effective for improving the responsiveness of websites and handling high volumes of traffic. It can handle thousands of requests per second, making it ideal for websites that receive large amounts of traffic. Additionally, Varnish offers a number of advanced features such as configuring custom caching rules, support for load balancing, and enabling advanced HTTP acceleration techniques such as Edge Side Includes (ESI). Thanks to these features, Varnish not only improves the performance, but also the scalability and reliability of websites, making it an essential tool for dynamic and high-traffic sites.
If you want to learn more about Varnish Cache we have written an exhaustive guide at this address Understand the logic of Varnish Cache
How does Vernice.js work?
Vernice.js was conceived as a middleware written in Node.js that sits between Ghost and Varnish. It runs independently of both systems, receiving webhook calls from Ghost, processing the received data, and interfacing with Varnish to perform selective cache purges.
When Ghost sends a webhook call following publishing, updating, or deleting content, Vernice.js receives this request. Ghost webhooks include a JSON payload containing relevant details, such as the URL of the modified content. Vernice.js extracts the URLs involved in the purging process, including the content-specific URL and the site's home page.
The middleware parses the JSON payload of the webhook to obtain urlObject.pathname e urlObject.host. This information is essential because Varnish cannot directly process a JSON payload. Vernice.js uses this information to properly format the purge requests that will be sent to Varnish.
The process works as follows: once the necessary details are extracted, Vernice.js sends an HTTP request of type PURGE to Varnish for the specified path and for the home page. This allows Varnish to selectively invalidate the cache only for affected URLs, ensuring that users always receive the most up-to-date content. Although the command sent is of type PURGE, Varnish can be configured to translate this command into a BAN, which is an effective method for selective cache cleaning.
An important thing to consider is that while you could theoretically connect Ghost webhooks directly to Varnish, in practice this isn't feasible. This is because Varnish cannot interpret and handle a JSON payload sent by Ghost webhooks. Vernice.js solves this problem by acting as an intermediary that translates the webhook information into a format that Varnish can understand.
What is a Custom Integration in Ghost?
Custom Integrations in Ghost allow you to connect Ghost with external applications and tools via webhooks, significantly extending the capabilities of the CMS. Custom Integrations are customizable configurations that allow Ghost to interact with external services automatically and reactively.
Webhooks are HTTP calls that Ghost sends to a specified URL when certain events occur, such as publishing, updating or deleting a post, creating a page, or adding a tag. When one of these events occurs, Ghost sends an HTTP POST request to the configured webhook, including the event details in the request payload.
These integrations allow developers to extend Ghost's functionality and automate various processes. For example, you can set up a webhook to send notifications to a chat service like Slack whenever a new post is published, automatically update a sitemap, or as in our case, purge Varnish's cache to ensure the latest content are served to users.
Custom Integrations offer a high degree of flexibility, allowing you to connect Ghost to virtually any service that supports the webhook API. This means you can create custom, automated workflows that respond in real time to events occurring on your Ghost site. These integrations are configurable directly from the Ghost admin interface, making it simple to add new webhooks and manage existing ones.
Thanks to Custom Integrations, you can improve operational efficiency and provide advanced functionality without having to directly modify the Ghost source code. This modular and scalable approach makes Ghost an extremely versatile and powerful platform for content management.
Configuring Ghost to Use Hooks via Web API
To properly integrate Ghost with the VERNICE.JS middleware and allow selective purging of the Varnish cache, you must configure Ghost to use Hooks via Web API. Below are the detailed steps to perform this setup from the provided image.
Log in to the Ghost admin panel and go to Settings (Settings), then select Advanced (Advanced). In the section Integrations (Integrations), select the tab Custom (Custom) and click on Add custom integration (Add custom integration).
Once you select the option to add a custom integration, fill in the required details to create a new integration. You'll need to enter a name for the integration, such as “Varnish Purge,” and a short description, such as “Purge Varnish at content modification.” After saving the integration, options to configure webhooks will appear.
On the Varnish Purge integration screen, click Add webhooks (Add webhook) and fill in the webhook details. Select the event that should trigger the webhook, for example “Post Published” or “Post Updated”, and enter the target URL of the server where the VERNICE.JS middleware is running. For example, if the middleware is running on the same machine as Ghost and on port 3000, the URL might be http://127.0.0.1:3000/webhook.
Repeat the process to add more webhooks that cover all events of interest, such as publishing, updating, and deleting posts, pages, or other content.
Webhook Configuration Example
Below is an example of what configuring a webhook for updating a post should look like:
Configuration Verification
Make sure Ghost is sending webhooks correctly when content changes are made. You can verify this by checking the vernix.js middleware logs. You should see log messages indicating that purge requests have been received for specific paths.
Middleware code VERNICE.JS
The middleware must be running on a Node.js server, ready to receive and process webhooks. Here is a summary of the code for VERNICE.JS:
/**
*
*
* (`-. ('-. _ .-') .-') _ ('-.
* _(OO )_ _( OO) ( \( -O ) ( OO ) ) _( OO)
* ,--(_/ ,. \(,------. ,------. ,--./ ,--,' ,-.-') .-----. (,------.
* \ \ /(__/ | .---' | /`. ' | \ | |\ | |OO) ' .--./ | .---'
* \ \ / / | | | / | | | \| | ) | | \ | |('-. | |
* \ ' /, (| '--. | |_.' | | . |/ | |(_/ /_) |OO ) (| '--.
* \ /__) | .--' | . '.' | |\ | ,| |_.' || |`-'| | .--'
* \ / | `---. | |\ \ | | \ | (_| | (_' '--'\ | `---.
* `-' `------' `--' '--' `--' `--' `--' `-----' `------'
*
*
*
* VERNICE.JS a Ghost <=> Varnish Middleware Selective Cache Purger and Daemon Cleaner.
*
* Copyright (c) 2024 - Managed Server S.r.l. - Licensed under AGPL 3.0 https://www.gnu.org/licenses/agpl-3.0.en.html
*
*
* Description:
* This Node.js middleware is designed to bridge the Ghost CMS with a selective Varnish Purging system.
* It listens for webhooks from Ghost related to content changes (like posts, pages, tags) and triggers
* selective cache purging in Varnish. This ensures that the content served to users is always fresh and up-to-date.
*
* Requirement:
* Node.js and express, axios, url, body-parser. Varnish Cache 3.0 or Higher. A Linux OS support systemd.
*
* Usage:
* Run this script in a Node.js environment. Configure Ghost to send webhooks to this middleware.
* The middleware will process these webhooks and send requests to the Varnish server to purge the cache as needed.
*
* Note:
* Ensure that webhooks in Ghost and Varnish cache purging rules are correctly set up for this middleware to function properly.
*
*/
const express = require('express');
const axios = require('axios');
const { URL } = require('url');
const bodyParser = require('body-parser');
const app = express();
const port = 3000;
console.log('\x1b[32m%s\x1b[0m',
" \n" +
" (`-. ('-. _ .-') .-') _ ('-. \n" +
" _(OO )_ _( OO) ( \\( -O ) ( OO ) ) _( OO) \n" +
" ,--(_/ ,. \\(,------. ,------. ,--./ ,--,' ,-.-') .-----. (,------. \n" +
" \\ \\ /(__/ | .---' | /`. ' | \\ | |\\ | |OO) ' .--./ | .---' \n" +
" \\ \\ / / | | | / | | | \\| | ) | | \\ | |('-. | | \n" +
" \\ ' /, (| '--. | |_.' | | . |/ | |(_/ /_) |OO ) (| '--. \n" +
" \\ /__) | .--' | . '.' | |\\ | ,| |_.' || |`-'| | .--' \n" +
" \\ / | `---. | |\\ \\ | | \\ | (_| | (_' '--'\\ | `---. \n" +
" `-' `------' `--' '--' `--' `--' `--' `-----' `------' \n" +
"\n"+
" VERNICE.JS a Ghost <=> Varnish Middleware Selective Cache Purger and Daemon Cleaner. \n"+
"\n" +
" Copyright (c) 2024 - Managed Server S.r.l. - Licensed under AGPL 3.0 \n" +
"\n"
);
app.use(bodyParser.json({ limit: '64mb' }));
app.post('/webhook', async (req, res) => {
if (req.body.post && req.body.post.current && req.body.post.current.url) {
const fullUrl = req.body.post.current.url;
const urlObject = new URL(fullUrl);
const postPath = urlObject.pathname;
const host = urlObject.host;
console.log(`Richiesta di PURGE per il percorso: ${postPath} e host: ${host}`);
try {
// Effettua una richiesta PURGE per la home page
axios.request({
method: 'PURGE',
url: 'http://127.0.0.1:80/',
headers: { 'Host': host }
}).catch(error => {
console.error('Errore durante la chiamata di PURGE per la home page', error);
});
// Effettua una richiesta PURGE per l'URL specifico
axios.request({
method: 'PURGE',
url: `http://127.0.0.1:80${postPath}`,
headers: { 'Host': host }
}).catch(error => {
console.error('Errore durante la chiamata di PURGE per l\'URL specifico', error);
});
res.status(200).send('Richieste di PURGE inviate');
} catch (error) {
console.error('Errore generico nel blocco try', error);
res.status(500).send('Errore durante l\'invio delle richieste di PURGE');
}
} else {
console.log('Payload del webhook non valido o mancante di informazioni cruciali');
res.status(400).send('Richiesta webhook non valida');
}
});
app.listen(port, () => {
console.log(`Server VERNICE.JS in ascolto sulla porta ${port}`);
});
By following these steps, you will be able to configure Ghost to send webhooks to the paint.js middleware thus ensuring that any content changes are immediately reflected in the Varnish cache, improving the performance and user experience of your site.
Keep the service active via a Systemd Unit.
To ensure that the Vernice.js middleware remains active and starts automatically when the system reboots, you need to configure a systemd unit. This is critical to ensuring that the service is always available to handle webhooks sent by Ghost. To do this, we will create a configuration file in /lib/systemd/system/vernice.service which will contain the following code:
[Unit] Description=Vernice.js systemd service for vernice.js Node Varnish Cache Purger After=network.target [Service] Type=simple WorkingDirectory=/root User=root ExecStart=/usr/bin/node /root/vernice.js Restart=always [Install] WantedBy=multi-user.target
Remember to adapt the paths based on the location of the paint.js file, in the Systemd Unit above it is assumed that it is running in the /root directory with root privileges.
The section [Units] defines the basic information and dependencies of the service. The parameter Description provides a brief description of the service. The parameter After=network.target indicates that the service should be started only after the network target (network.target) has been achieved, ensuring that the necessary network resources are available.
The section [service] defines how the service should be performed. The parameter Type=simple specifies that the service is simple and started directly from the command indicated in ExecStart. The parameter WorkingDirectory=/root set the service working directory to /root. The parameter User=root specifies that the service should run under the root user. The parameter ExecStart=/usr/bin/node /root/vernice.js indicates the command to start the Vernice.js middleware; in this case, Node.js is used to run the script vernice.js located in the directory /root. The parameter Restart=always configure the service to automatically restart if it ends unexpectedly, thus ensuring greater reliability.
The section [Install] defines how and when the service should be started. The parameter WantedBy=multi-user.target specifies that the service should be started as part of the target multi-user, which is one of systemd's standard execution levels and includes multi-user mode without a graphical interface.
After creating the configuration file, you need to activate and start the service with the following commands:
sudo systemctl daemon-reload sudo systemctl enable vernice.service sudo systemctl start vernice.service
These commands will reload the systemd services configuration, enable the service to start automatically on boot, and immediately start the Vernice.js service. With this configuration, Vernice.js will always be active and ready to handle Ghost webhooks for Varnish's selective cache purge.
Varnish setup for Ghost
At the Varnish configuration level, by company policy we do not go into the configuration of the configuration and the related VCL, adopting a specific and customized version extended with Inline C, however the configuration can be applied simply by making sure that the Varnish recv block can be capable to correctly manage the PURGE command by selectively BANing the URL.
Below we can see a short snippet of the configuration in which there is control of the requesting IP (which must correspond to either the machine IP or the localhost IP) which then proceeds to BAN the URL that is passed by the paint.js middleware .
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
if (req.request == "PURGE") {
if ((req.http.X-Forwarded-For == "91.107.202.139, 127.0.0.1") || (req.http.X-Forwarded-For == "127.0.0.1")) {
ban("req.url ~ ^" + req.url + "$ && req.http.host == " + req.http.host);
}
else {
error 405 "Not allowed.";
}
}
If you need to install and configure Varnish we could provide commercial advice on the matter and possibly also extend the functionality.
You can find any upgrades, forks, and fixes at the official GitHub repository at https://github.com/MarcoMarcoaldi/Vernice.js
