Table of contents of the article:
Some migrations are difficult because the infrastructure is complex. Some migrations are difficult because there's a lot of data, because an application was poorly developed, because a database is huge, or because you need to work on legacy systems that haven't been touched for years.
And then there are migrations that become difficult for a reason that, frankly, after so many years of this profession we continue to struggle to accept: the lack of collaboration between suppliers.
This article tells a real case we dealt with in recent days.
We won't name the client and will call his two domains example.com and exampletwo.com , because the point of this article is not the client.
The point is us suppliers.
The point is to understand what should happen when a company decides to change technology partners and, above all, what should never happen.
It all started with two very slow WooCommerces
The client contacts us because he manages two WooCommerce sites with significant web performance issues.
High response times, an unresponsive backend, and an overall situation that made it necessary to evaluate a different infrastructure, which we discussed yesterday on our blog in this post.
After the initial checks, the decision was made to migrate the services to Managed Server Srl.
An absolutely normal situation.
Anyone who works in hosting knows perfectly well that clients come and clients go. It happens to us, it happens to our competitors, and it will continue to happen.
There is no perfect supplier for every project and for every stage of a company's life.
A customer may choose another hosting provider because they're looking for different services, need more specialized expertise, change budgets, change management, or simply want to try something new.
It's part of the market.
In this specific case, in addition to the websites, the email accounts associated with the two domains also had to be transferred.
And this is where a technically ordinary migration turned into something completely different.
The request was very simple: do not touch anything else.
To properly organize a migration of this type, you don't need anything particularly exotic.
We had essentially asked for two things: to deliver the backup of the sites and to change the nameservers of the.
Nothing else.
We didn't ask for the accounts to be deleted. We didn't ask for the services to be immediately suspended. We didn't ask for the mailboxes to be deleted.
We hadn't asked for it, and the customer hadn't asked for it either.
Conversely, during a migration it is essential that the old infrastructure remains available for the time strictly necessary to complete the synchronizations.
This is especially true for email.
A site can be copied at a given point and then realigned. Email, on the other hand, is constantly changing: new messages arrive, users reply, move and delete emails, send attachments, create drafts, and constantly change the status of their folders.
A mailbox is a living dataset.
For this reason, a successful IMAP migration is typically done in multiple steps, keeping the source server accessible until the cutover is complete.
The expected strategy: imapsync
To transfer the mailboxes we had set up a normal migration via imapsync.
Mail server administrators are familiar with this tool. Its operating principle is relatively simple:
SERVER IMAP SORGENTE | | IMAP v imapsync | | IMAP v SERVER IMAP DESTINAZIONE
imapsync authenticates on the source mailbox, reads folders and messages, and replicates them to the destination.
One of the great advantages of this methodology is that, from a migration point of view, it is not necessary to know in detail the format in which the source server physically stores the mail.
The server may be using Maildir, mdbox, or another backend supported by its IMAP service.
From the migration client's perspective we simply see something like this:
INBOX Sent Drafts Trash Archive spam
and, of course, the related messages.
It is the source server that is responsible for translating its physical storage into the logical representation exposed via IMAP.
There is also another fundamental feature: imapsync can be performed multiple times.
We can then perform an initial synchronization of a 20 GB mailbox, wait for it to complete, and then run the process again a few hours before the final switch.
The second sync doesn't necessarily have to transfer all the data again. Its main purpose is to recover the delta , that is, everything that has changed between the first copy and the final switch.
This is a very common methodology in email migrations and allows you to significantly reduce the risk of losing messages during the transfer.
Then, not even 24 hours later, the accounts are suspended
The old infrastructure was managed by a company operating in the Web Development, SEO and Web Marketing sectors.
And it was at this point that something happened that, professionally, left us very disappointed.
Despite the request not to intervene further on the services during the migration, less than 24 hours later the accounts were suspended , effectively making the IMAP mailboxes required to complete the synchronizations unreachable.
The technical effect was immediate:
imapsync | X | SERVER IMAP SORGENTE NON PIÙ ACCESSIBILE
From that moment on we could no longer perform the normal mailbox realignment.
And, most importantly, we could no longer recover messages via IMAP that arrived after the last available copy of the data.
On this point we want to be very clear: we are not discussing the intentions of those who made this decision.
We certainly didn't order it, nor did the customer who tried unsuccessfully to reactivate the account.
We are describing the objective technical effect we encountered: the account suspension prevented the IMAP procedure from completing as planned.
From that moment on we were forced to completely change strategy.
Two certified emails, emails, WhatsApp, and phone calls
The part that disappointed us the most wasn't even the technical difficulty.
Technical problems get solved. That's our job.
What we found much more difficult to understand was the impossibility of establishing a direct technical comparison with the previous supplier.
We sent two certified emails , wrote via email , communicated via WhatsApp and tried to call.
And the customer did too.
Despite repeated attempts, we have not been able to establish a direct technical dialogue that would allow us to coordinate the migration.
And this is probably the aspect on which we would like to invite all of us operators in the sector to reflect.
Because a phone call would have been enough.
Five minutes.
“When are you done with the mail?”
“Tomorrow at 18pm.”
“Perfect, let’s leave IMAP on until tomorrow evening.”
End.
No wars. No special procedures. No need to manually recover email storage from a backup.
We did have a cPanel backup though
Luckily, we had a recent enough cPanel backup.
At that point we began to study the possibility of directly recovering the email contained in the backup.
And this is where the story becomes technically interesting.
When we opened the mailbox structure we noticed that we were not looking at a normal Maildir.
We found ourselves in front of Dovecot mdbox.
Maildir and mdbox are not the same thing
On our infrastructure, mail is handled through Dovecot using Maildir.
A traditional Maildir has an easily recognizable structure:
Maildir/ ├── cur/ ├── new/ ├── tmp/ ├── .Sent/ ├── .Drafts/ ├── .Trash/ └── .Archive/
The basic concept is relatively simple: a message is essentially the same as a file.
If we open the directory cur/ We find hundreds, thousands, or, in the most significant cases, millions of files. Each of those files represents a message.
mdbox works completely differently.
In the cPanel backup we found structures of this type:
storage/ ├── dovecot.map.index ├── dovecot.map.index.log ├── m.1 ├── m.2 ├── m.3 ├── m.100 ├── m.305 └── ...
and, at the same time:
mailboxes/ ├── INBOX/ │ └── dbox-Mails/ │ ├── dovecot.index │ ├── dovecot.index.cache │ └── dovecot.index.log ├── Sent/ ├── Drafts/ ├── Trash/ ├── Archive/ └── spam/
Here the paradigm changes completely.
In mdbox a file like:
m.305
This does not represent email number 305.
It's a container.
It can store multiple messages. Dovecot then uses maps and indexes to determine where each message is located and which mailbox it belongs to.
Among the essential files we find elements such as:
dovecot.map.index dovecot.map.index.log dovecot.index dovecot.index.cache dovecot.index.log
Consequently, it was not possible to trivially take:
m.1 m.2 m.3 ...
and copy them into the new Maildir.
It wouldn't have made any sense from a storage format standpoint.
From a migration to a true storage recovery
At this point what was supposed to be a normal IMAP migration had become a full-blown recovery operation.
We needed to make Dovecot correctly interpret the old mdbox storage and logically rebuild the mailboxes.
We then set up a separate work environment.
The first rule in such an activity is very simple: do not work directly on the only available copy of the backup.
Individual mailboxes were copied to temporary directories and all operations were performed on the working copies.
We then used doveadm to test whether Dovecot was actually able to read the storage.
After managing the UID, GID, and permissions of the technical user used for the conversion, we were finally able to query an mdbox mailbox.
The result was something like this:
Archive Sent Drafts Trash spam INBOX
That was the turning point.
This meant that Dovecot was correctly interpreting:
- the files
m.*containing the messages; - the file
dovecot.map.index; - the indexes of individual mailboxes;
- the relationship between messages, physical storage, and logical folders.
The backup was usable.
The next problem: mdbox and Maildir have different layouts
At that point we could read the source correctly.
It remained to be transformed.
We then started using doveadm e dsync To convert from mdbox format to Maildir format:
Dovecot mdbox | | dsync v Dovecot Maildir
Here too, however, it was not enough to simply indicate a source and a destination.
One of the first problems encountered was with the virtual mailbox hierarchy separator.
Essentially, the source mdbox storage and the destination traditional Maildir++ did not use the same representation of the mailbox hierarchy.
The result was an error like this:
Mail locations must use the same virtual mailbox hierarchy separator
It was therefore necessary to make the destination namespace and layout compatible using a Maildir with:
LAYOUT=fs
finally obtaining a coherent structure:
utente/ ├── cur/ ├── new/ ├── tmp/ ├── Archive/ │ ├── cur/ │ ├── new/ │ └── tmp/ ├── Drafts/ │ ├── cur/ │ ├── new/ │ └── tmp/ ├── Sent/ ├── Trash/ └── spam/
At that point we again had real messages in Maildir format , usable on the new infrastructure.
Even the shares wanted to join the party
Another problem appeared during the conversion.
The Dovecot of the machine used for the procedure had a backend quota-dict.
dsync, while writing the converted messages, it also attempted to update the mailbox quotas:
quota-dict: Quota update failed Quota is now desynced
In normal production server operation this behavior would make perfect sense.
During an offline conversion, however, no.
We weren't interested in updating the quota for a temporary mailbox in real time. We were interested in correctly retrieving messages.
We have therefore excluded the quota plugin from the invocations used for the conversion:
mail_plugins=''
and recovery is complete.
Once the mailboxes have been placed in their final destination, the quotas can be reconstructed via Dovecot with a normal recalculation procedure, for example:
doveadm quota recalc -u [utente@esempio.com](mailto:utente@esempio.com)
followed by the relevant verification:
doveadm quota get -u [utente@esempio.com](mailto:utente@esempio.com)
What was initially supposed to be a simple:
imapsync sorgente → destinazione
had now become:
backup cPanel | v analisi storage | v riconoscimento mdbox | v copia di sicurezza | v lettura indici Dovecot | v ricostruzione mailbox | v gestione UID/GID | v gestione namespace | v conversione mdbox → Maildir | v gestione quota | v importazione | v ricalcolo quote
All this to achieve what, under normal conditions, we could have transferred through a normal IMAP synchronization.
The backup saved us, but a backup is a photograph
Fortunately, we were able to recover the email from the backup.
However, there is a problem that no amount of system engineering expertise can solve: a backup can only contain what existed at the time it was generated.
The available backup was up to date as of August 6th.
When we had to carry out the recovery it was already August 10th.
Backup disponibile: 6 agosto Data del recupero: 10 agosto Gap temporale: circa 4 giorni
A backup is a photograph.
You can know Dovecot in every detail, you can rebuild indexes, convert storage, repair permissions, and correctly interpret mdbox, but you cannot recover from a snapshot of August 6th something that happened on August 7th, 8th, 9th, or 10th.
If that data isn't present in the backup, it simply doesn't exist within that dataset.
And that's exactly why we planned to use it. imapsync.
We could have used backup or an initial sync to transfer the bulk of the data and then queried the old server again to capture everything that had changed in the meantime.
In other words: the delta.
But if the source server is no longer accessible, that delta cannot be acquired.
The concrete result was therefore a gap of approximately four days of unavailable mail history within the backup used for recovery.
And this is probably the part of the whole story that leaves us most saddened.
It's not a question of being competitors
We and the other company are, at least in part, operators in the same market.
These are therefore entities that, in some respects, can be considered competitors.
But being competitors doesn't mean being enemies.
Over the years we have received dozens and dozens of requests relating to migrations of Managed Server customers to other providers.
It happens.
When a customer decides to leave, the professionally correct thing to do is to hand over what is needed and allow the new supplier to work.
If a systems engineer called us tomorrow and said:
"I'm migrating one of your former clients. Can you leave IMAP available for another 48 hours to complete the delta?"
the technically sensible answer should be:
"Certain."
Not because we are good.
Because it's technically correct.
Because the data belongs to the customer.
Because on the other end there is a professional who is trying to do his job.
And because the day after tomorrow we might be the ones having to call him.
The customer must not become hostage to rivalry between suppliers
There is a principle that we should remember more often in our industry.
When a customer changes supplier, the switch should be as transparent as possible for the customer.
There should be no need to involve him in technical discussions between two companies.
You shouldn't be acting as a switchboard operator between two IT providers.
You shouldn't have to figure out what an MX record, nameserver, mdbox, Maildir, rsync, imapsync, or incremental sync is.
We should talk to each other.
“What do you need?”
"This."
"Until?"
"Tomorrow."
"All right."
In many cases this would actually be enough.
Our deep professional disappointment
What we feel at the end of this story is above all profound professional disappointment.
Not because we had to work harder.
We chose to be systems engineers also because we like solving complex problems.
And, from a purely technical point of view, turning a cPanel backup containing mdbox mailboxes into perfectly readable Maildir mailboxes was even an interesting problem to tackle.
The disappointment comes from the fact that it would not have been necessary to do any of this.
There would have been no need to rebuild the storage.
There would have been no need to work on the Dovecot indices.
There would have been no need to resolve namespace incompatibilities.
It would not have been necessary to convert mdbox to Maildir.
There would have been no need to manage quotas offline.
And, most importantly, it would have been possible to perform the last message synchronization.
It would have been sufficient to leave the accounts available for the time necessary to complete a migration already in progress.
An appeal to colleagues in hosting and systems engineering
This article isn't just meant to be a story of a bad experience with a specific supplier.
It is intended above all as an appeal to all companies working in hosting, web development, web agencies and systems engineering.
Let's work together.
Even when we lose a customer.
Even when the new supplier is a competitor.
Even when we think the customer is making the wrong choice.
We can explain it to him. We can be against it. We can even be convinced he'll come back to us after three months.
But when a customer decides to migrate, we make sure they can do so in an orderly manner.
- If the new provider asks for an additional 48 hours of IMAP access, let them have it.
- If it asks for a MySQL dump, we provide it.
- If it asks not to power off the old virtual host until DNS propagation, we wait.
- If you need to agree on a cut-over window, let's talk on the phone.
- and above all if the paying customer does not ORDER to cease the service, do not act on your own.
Because in the end, this is our job.
Don't put obstacles in the way.
Solve problems.
We finally made it
Despite everything, we managed to recover the email in the backup.
We interpreted the mdbox storage, verified the Dovecot indexes, rebuilt the mailboxes, converted the messages to Maildir and prepared the mailboxes for the new infrastructure.
Available mail has been recovered up to the consistency of the August 6 backup.
The regret remains for the subsequent gap, which a normal IMAP synchronization could have avoided.
And above all, one question remains.
Was it really necessary to come to all this?
We don't think so.
For this reason, after this experience, the message we want to leave is extremely simple:
Dear "colleagues," we may be competitors. But let's collaborate and not put spokes in each other's wheels.
We profit from it.
Those who come after us will benefit.
But above all, the customer benefits, as he or she should be the only person who doesn't have to pay the consequences of a lack of collaboration between two suppliers.
