Table of contents of the article:
When writing on a company blog, there's always a tendency to employ that corporate/institutional style of cold, uninspiring language, recounting everything and nothing. Unlike big American corporations and small businesses that mimic them, we want to share empathetic, real-life professional experiences that will allow you to experience real-life situations firsthand and learn from others. Because in a world of companies, customers, suppliers, and mere self-employed individuals, there are people's stories that deserve to be told.
Today I want to talk to you about backup and data security , telling you what I have and what we have to say about an issue that is far too underestimated.
If something can go wrong, it will go wrong.
This pseudo-scientific axiom, better known as Murphy’s Law, has been the cornerstone of organizational processes regarding security and backup management.
With this awareness, over the last 6 years we have evaluated and implemented robust and proven backup solutions to ensure the integrity of customer data.
Operating with the old Dreamsnet.it brand since 2005 and with previous systemic experiences since 2000, up to now we had always used an incremental snapshot backup solution that allowed a selective restore of both individual files and entire images. At the database level, for example, we used (and still use) tools that allow you to take snapshots quickly and respecting the logic of hot backups and integrity, that is to backup the DB without having to turn it off and making it always available the service even if it happens at 4 am (maybe the users are asleep, but the search engines index).
In short, not exactly the last of the class, if we think that today we have famous Italian hosting that still use the ancient and ancestral mysqldump. Really unbelievable.
In thirteen long years, in short, and after several hundred restored backups, no customer has ever lost a single file of his projects.
In the summer of 2018, something new, unexpected, and shocking happened that would forever cast doubt on ourselves and everyone else after that terrible ordeal.
It was a sunny day, warm but not muggy. I was wearing a short-sleeved shirt and shorts, with my laptop slung over my shoulder. I had just finished lunch at the riverside park that runs alongside my house in Arad, Romania. I remember walking back to the office at the Arad Business Center , riding along the city's beautiful bike path, which was very busy at that hour with everyone else on their lunch break, like me.
The view of the still river inspired peace and tranquility, the sight of people relaxing on the benches conveyed calm and positivity, I was literally immersed in that moment of bliss when the cell phone started ringing , having been diverted directly from the main office.
I return to planet earth, the one made of other people's problems to be solved (after all, work generally has this purpose, doesn't it?) and I respond with a warm
Hello, I'm Marco from Managed Server, how can I help you?
A male voice replies, it was a male, a boy roughly my age, between 30 and 35 I would have estimated by ear. We immediately overcome the formalities of the Italian language and put ourselves at ease by calling each other immediately (by the way, on the net and on social networks we call each other TU says Netiquette did you know?)
He begins, somewhat nervously, to tell me about backups, restores, failed restores, lost data, and data recovery. Too many abstract and confusing concepts, too much fragmented input, and a lot of confusion in my head. I don't understand.
Who is the one who is phoning me? Is he our customer?
What happened?
Talk about backups, but do you need to restore a backup? Lost a backup? Don't have a backup?
Could it be one of those kids who can't tell the difference between a washing machine and a VCR?
I immediately stop this flood of random terms and phrases and ask you to calmly explain everything that has happened from the beginning, and ask me what your needs were.
Let's start again much more calmly and finally establish a discussion made up of sentences that make sense and above all logically connected.
In simple terms
This guy, hosted by a well-known French hosting company, said that the night before he had made a mistake while putting some FTP files into production and, out of frustration at solving the problem, he had decided to do a “tabula rasa”, that is, delete everything and restore a backup from the day before.
Therefore, he is about to connect with his FTP client, delete all the folders of his site and once finished, from the customer area, execute the backup restore procedure from the comfortable Web interface.
Four clicks and you're done!
After a few minutes of progress of the task, a message announced the completion of the restore operation.
Oh, joy, jubilation, and exultation! What better news?
What happened is easy to understand and as you have already guessed, the backup was damaged or rather EMPTY. The restore wizard had restored exactly ZERO FILES leaving the destination directory completely empty, instead of the files from the previous day.
He doesn't get discouraged, doesn't give up, and tries again. Maybe maybe something went wrong.
Same procedure, same archive, same restore message. Then you check with the FTP client and… EMPTY. Again no file. Nothing, Nada, zero, nisba.
Return to the interface and select the backup of the day before, same restore procedure. Same successful completion message, same result. EMPTY.
Select the backup of the day before the day before. Restore. EMPTY
And so on with the backup of the day before the day before the day before yet. 3 days before, 4 days before, 5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30, XNUMX days before.
Same result: EMPTY.
The archives to be restored had run out, and the conclusion was only one: none of the 30 backups up to 30 days before had been possible to restore. This person had no way to restore backups of him, site of him feeding him.
The problem had become something really big as it was apparently without solution.
At that moment I felt a mix of emotions between them. Sadness mixed with joy.
You know when you experience some bad event directly or indirectly in which out of empathy you feel sadness about what happened but deep down also a pinch of joy in knowing that that bad thing didn't happen to you? Here, just that feeling, that mix of emotions at the antipodes, which collide with each other.
I was glad he wasn't a customer of ours; I wouldn't have been able to handle a situation like that, where the backup service exists but none of the archives can be restored. What could I say to a PERSON who lives off the site and supports their family, who was on the phone with me at that moment?
I immediately thought that this moral ethical problem, that sense of responsibility would not have affected the huge hosting company of which this person was a customer in the least.
In the end it was clear how the story would go: this person would have spoken to the telephone support, some guy put there a few coins a month, would have forwarded the problem to the Italian technical department who would have forwarded it to the French one and one of his technicians . The technician would probably have noticed that something was wrong and would have sent the message back to the Italian technical support who would probably have stated the lack of data and possibly asked for a refund or open a legal dispute.
Legal dispute. In Italy. This would be laughable if it weren't for the tragic situation. It makes me cry, however, thinking about the signing of a contract and an SLA (written in such tiny fonts they should be illegal), in which the supplier indemnifies itself from any such damages and any compensation claims.
It would have been easier to climb Everest barehanded. (Okay, that's not Everest in the photo, but it gives you the idea.)
The company's CEO would never have even known about the story of this desperate father who lost five years of work. One of many among millions of customers, a drop in the ocean. What does a problem like this mean for a company that generates hundreds and hundreds of millions in revenue each year with millions and millions of customers? Absolutely insignificant. Unfortunately.
What could we do?
Nothing. Faced with that complex situation and absolutely beyond mine and our margin of maneuver (the supplier was another, we didn't know what backup system he used, what the storage system could be, how to have access to the media, and why had this bad mishap happened), what could we do?
The most correct thing to do was obviously to ask for a comparison with their technical support to understand if there was any possibility of recovery, and in the meantime to look if CASOMAI had a local backup on your PC maybe a few days or months before.
The first suggestion was unsuccessful; support simply stated that there were no files in the backup, even though it had been created correctly. This statement was reiterated by their top-notch French technical support team about two weeks later, who simply dismissed the situation with " there's nothing there ," an absolutely grotesque and unpleasant situation.
The advice to find a local backup, however, is a little better, given that with a lot of dedication and digging through his PC's files, he actually managed to find and restore a backup from a few months earlier. Although it wasn't the ideal way to repair such damage, it still allowed him to get back on track and avoid the inevitable bankruptcy that would have occurred if this had not been the case.
All's well that ends well!
The lesson we have learned
In all this ugly history, our role has been absolutely spectator. Absolutely irrelevant, in short, for the outcome of the story fortunately resolved if not for the best, certainly not for the worst.
All this, however, gave us the opportunity to learn some important considerations that had previously been overlooked. Essentially, we asked ourselves the following questions:
1. Why can't what happened to them happen to us?
Why should a backup system fail only happen to other hosting providers and not to us?
That would be hypocritical, right? It's like saying we could easily avoid wearing seatbelts while driving because accidents only happen to other people, right? Instead, the most correct and appropriate answer is that up until that memorable summer day, the only reason it hadn't happened to us was pure chance, pure luck. The fact, in short, that a snapshot backup system, and a highly advanced one, had never generated corrupt backups. A remote possibility, certainly, unlikely, but not impossible, as we saw that very day.
2. How much does data loss affect people's lives?
Losing a website or data can spell the end of a business. This means creating an economic damage on people's lives, probably putting them in a position not to be able to afford goods of primary necessity. This cannot and must not happen, at least it must not happen through our fault, neither as a cause nor as a contributing cause.
3. How much does data loss affect our company?
The loss of a customer's data will most likely lead to legal disputes. It's irrelevant whether you're right or wrong, or whether you need to review the various liability indemnities and SLAs you've signed. Clear legal obligations exist, such as those imposed by the new European GDPR law, which would expose us to various omissions and therefore subject us to fines and compensation. It's better to invest in data security by allocating 10% of your turnover to monitoring systems, RAID systems, redundant storage, and multiple backups, rather than risk court cases, litigation, compensation, and fines.
What did we do then?
Having understood the three points above loud and clear, we “simply” added a secondary one to the already functioning backup system, in turn redundant on a C14 data storage system with military grade certification in France.
In short, if previously on our RAID1 systems we had only one backup that flowed into a RAID5 storage area, today we have three backups with two different technologies that flow into three different RAID5 storage areas and in turn one of these is mirrored on a secure redundant military grade C14 anti-atomic storage service in France.
In short, in this modus operandi, an unfortunate customer who needs to restore a backup can rely on a remote backup which, if it doesn't work (a very remote hypothesis), could rely on the second backup system, which in turn, if it doesn't work ( hypothesis itself very remote) could rely on mirroring with Rsync on another storage.
If there is a major attack (let's assume a nuclear explosion) on the datacenter, we would still have a weekly mirroring on the C14 anti-atomic storage in France.
In short, the possibility of losing customer data has really been reduced to an almost impossible limit. More than these precautions do not currently exist to our knowledge in Italy but also internationally, seen and ascertained that even very large companies with turnovers of hundreds of millions of euros continue to use a single and unique backup as a definitive solution to protect themselves itself and its customers.
Internally, as we always have, we will continue to offer the Backup service as part of our package. There's no way it can be an additional, value-added service with a separate price. We offer backup as standard; we will ensure that our customers are protected to the best of our ability , regardless of whether they're optimistic that nothing will ever happen to their data, or that it's useless.
In short, atomic bomb proof!
