Moving a server without stopping it: on-line migration via reloca.to

Why a night window is not enough for everyone

A classic migration asks the old server to go quiet. You boot it into a Live OS, copy its disks over the network, assemble the system on the new hardware, check it and only then turn it on. For a server with a terabyte of data that is anything from half an hour to several hours, and all that time the site is down, the database does not answer and mail piles up at the senders.

Some customers agree such a window for the night and move without drama. Others put the move off for years. One runs a shop, another runs billing, a third simply has nobody willing to sit until morning watching a console. We heard this often enough to try removing the window itself.

What we could borrow from virtualization

Virtualization solved this long ago. VMware's vMotion and KVM's live migration carry a running machine to another host: memory is copied in several passes while the guest keeps working, and at the end the guest stops for a fraction of a second, catches up on the last piece and wakes up on the new host.

Dedicated servers cannot work that way. The cpus, chipsets and network cards differ, and moving RAM together with the state of every process is impossible, so a reboot stays in the picture whatever we do. The main part of the idea still carries over: copy the disks while the system is running, re-send only what changed in a few passes, and stop the server at the very end and only briefly. We call that mode on-line relocation; the classic move, with a Live OS on both machines, is off-line relocation.

Copying a disk that is being written to

In on-line mode the old server stays in its own system. The tool drops one static binary, reloca-agent, onto it, runs it as root and talks to it over ssh. No packages appear on the donor, no daemons, and nothing is written to its disks.

The agent does two things at once. It reads the block devices directly, bypassing the cache, and sends them as frames over the network. And it watches every write to those devices, so it knows which parts of the copy are already stale.

It watches in one of two ways, depending on the kernel. Where eBPF is available the agent assembles a tiny program in memory, attaches it to the block_bio_queue tracepoint and marks every 64 KiB chunk somebody wrote into in a bitmap. Where eBPF is not available, and CentOS 7 with its 3.10 kernel keeps turning up among the systems people move, it turns on ftrace and reads the same stream of events as text from trace_pipe. The second way costs more, and it works on everything we have tried, back to 2013.

The first pass reads the volumes whole. Empty space does not travel: the agent recognises a block of zeroes and sends a short record saying “so many zeroes here” instead of the data, while the recipient zeroes the whole disk with a single ioctl at the very start, so it has nothing to write at all. On a volume with 2.7 GiB used out of 12 that saves both traffic and minutes.

Passes and convergence

After the first pass the agent closes the tracker's epoch and looks at what has gathered. From then on only the changed chunks cross the network. Every pass is shorter than the one before it, because less time means fewer blocks spoiled, so the delta shrinks from pass to pass.

The trick is stopping at the right moment. The agent works it out like this: at the link rate it has already measured, the remaining delta must take no longer than the downtime you are willing to accept. That is ten seconds by default, -o converge=10. As soon as a delta fits, the final sequence begins.

A few seconds of freeze

A copy of a live filesystem is always slightly inconsistent: while the beginning was being read, somebody rewrote the middle. So the last delta is taken with writing stopped.

What stops it is fsfreeze, the FIFREEZE ioctl. The kernel flushes dirty pages, freezes the journal and holds back everyone trying to write. Processes do not die and do not get errors, they simply wait inside their write for a few seconds. Reading keeps working, so nginx still serves static files and an ssh session does not drop.

The agent freezes every mounted filesystem of the donor at once, deepest mountpoint first. What cannot be frozen it skips and says so in the log: an ESP on vfat does not support the ioctl, and on a booted system nobody writes to the boot partition anyway. Then the last delta passes and the donor stays frozen until you confirm. If something is wrong on the new server, the rollback is one fsfreeze -u from the console, after which the old server carries on as if nothing had happened.

On our test bench with Debian 12 and Rocky 9 the freeze window comes to between 1.7 and 2.4 seconds over a 10 GbE link. That window is the whole downtime, up to the reboot of the new server.

When the deltas stop shrinking

This does not always converge. If the server writes to disk faster than the link carries the data away, the next delta will be no smaller than the last one, and you can wait forever.

So the agent measures both rates: how many megabytes a second go onto the wire, and how many megabytes a second are dirtied on the donor. From the ratio it computes directly how many passes are left until the window you asked for and how many minutes that will cost, and it says so out loud:

the donor writes 20.4 MiB/s, the link carries 14.9 MiB/s (1.37x): the deltas
cannot shrink, this will not converge

When the prediction looks optimistic, the agent does not stop at the pass limit. If the arithmetic says three more passes would reach the window and cost twelve minutes, it raises the limit itself and keeps copying as long as it fits the patience budget (-o patience=1800, half an hour by default). If it does not fit, the agent stops and offers the window it has.

And if even that window is too long, the run refuses to freeze the donor at all, and explains what would change the answer:

the shortest freeze on offer is 180 s, above the 100 s this run was allowed
(--max-est): nothing is frozen, the donor is untouched
what changes it: a faster link, less writing on the donor while it copies,
or -o max-est=180

By that point the new server's disk is already written, while the old server has not been stopped for a second, so the move can simply be repeated at night, when there is less writing.

Compression that picks itself

Compressing frames on the donor pays off exactly as long as its cpu keeps ahead of the link. On 10 GbE one core running zstd quickly becomes the bottleneck. On a 100 Mbit link between data centres that same core sits idle.

This used to be an option you had to guess. Now the agent measures it too. It knows how many seconds it spent compressing and how many bytes came out, which is the throughput of its own compressor in megabytes a second, and it compares that with the speed of the wire. While the cpu is twice as fast as the link and the data actually compresses, the level climbs (1, 3, 5, 7). As soon as it stops keeping up, the level comes back down and that ceiling is not tried again.

compression: donor 621 MiB/s at zstd:1, link 11 MiB/s, ratio 1.56x -> zstd:3

On CentOS 7 over 100 Mbit the whole move took 823 seconds with compress=auto, against 953 seconds pinned at level 1 and 932 at level 3.

The pre-check: a rehearsal on a snapshot

Off-line the order is simple: copy, fix the copy up in a chroot, verify, reboot. On-line the fix-up and the verify land on a copy that is still catching up with the original, and you want to learn that the system will not boot before the old server has left the network.

So by default the donor is frozen twice. After the first freeze the tool takes a dm-snapshot of the copy and thaws the donor. While it serves its users again, the fix-up and the verify run in full on that snapshot: an initramfs with the new controller's drivers is built, grub is installed, the boot entry, fstab and network are checked. The snapshot is dropped, the deltas catch up, the donor is frozen a second time, and only now does the real switch happen.

It costs one extra freeze window. On a fast link both freezes are about a second. On 100 Mbit it is 100 seconds before the pre-check and another 87 before the cutover, and there -o precheck=0 saves noticeably, at the price of the fix-up and the verify running for the first time after the last freeze.

What the bench showed

All the numbers below come from the same migration: CentOS 7.9, root on XFS, 2.7 GiB of data, a load generator writing about 20 MB/s to the disk. The link between the machines is shaped with netem, so the latency and the loss are real.

The third row is the case where the server writes faster than the link carries. The agent predicted a window of 98.8 seconds and the real one came out at 100.1, so the estimate the decision rests on is sound. On a link like that it is better to move in the quiet hours, or to take the off-line mode and an honest maintenance window.

What on-line relocation does not do

• It does not move RAM. Processes start again when the new server boots, as after any reboot.
• It does not keep the IP address: the new server lives in our network with an address of its own, and DNS has to be switched. That is true of any move between providers.
• It does not work without root on the old server, or without a kernel that can track writes.
• It does not handle btrfs, ZFS, LUKS or NTFS. What is supported is ext2/3/4 and XFS, including on md RAID and LVM.
• It does not avoid rebooting the new server: that reboot is the switch.

Which mode to take

The tool, the phases and the plan are the same in both modes. The mode is detected from where the old server answered: the Live OS, or its own system.

Where to start

1. Order a server at XServer and boot it into the Live OS: client panel, the server, boot menu, Live OS.
2. Nothing has to be done on the old server, it keeps running. All you need is root access over ssh.
3. Log in to the new server, install the tool and look at the plan in a dry run. Nothing is written until you have read what is on the disk and typed its name and size off the screen.

   ssh -A root@NEW-SERVER
   curl -fsSL reloca.to | bash
   reloca.to -n root@OLD-IP      # dry run: the plan and every command
   reloca.to root@OLD-IP         # the whole move

4. Keep the new server's console open for the switch: the first boot is only visible there.

Every option with its default is collected in the reference on reloca.to, together with the phases and the exit codes. If the layout is unusual, the OS is old or the verify did not pass, write to XServer support: the old server is untouched through all of this, so there is nothing to lose.