XServer Live OS: a rescue system that knows what to do

XServer Live OS is a system for servicing a server that boots over the network and lives entirely in RAM. It writes nothing to the disks unless you ask it to. You start it from the control panel: just pick Live OS in the server's boot menu.

Live OS comes in two stages:

• Base Live OS is a small BusyBox system. It boots in seconds and opens a menu on the server's screen right away. It already has everything for diagnostics, repair and booting the installed OS.
• Full Live OS is Debian 13 with systemd and apt. About 150 MB is downloaded straight into memory from a single menu item. You need it when you run out of tools: to install a package, fix Windows or move a server.

You can switch between the stages as many times as you like without rebooting the server: the live command moves to the full one, base returns to the menu.

Work starts in the base Live OS: from there one command takes you to the full one, and one menu item starts your system from the disk.

Instead of an empty command prompt, Live OS starts with a menu. Each item is chosen with a single key, and two lines below it say exactly what will happen. Anything that can change something asks for confirmation with Yes and No buttons.

While you are reading the menu, Live OS is already collecting hardware details in the background. That's why the first two items usually open instantly, without waiting for the disks to be polled.

Diagnose checks everything between powering on the server and a running system, and gives a verdict for each item: OK, WARN (worth a look) or FAIL (this is what stops the boot). Next to each problem it says how to fix it.

Diagnostics only read. File systems are mounted without replaying the journal, the software RAID is assembled read-only, and everything is unmounted when it finishes. You can run it as many times as you like, with no risk.

For Linux it checks:

• the installed system's boot mode against the firmware mode (BIOS or UEFI);
• GRUB in the MBR or the boot loader on the EFI partition and its entry in the firmware;
• grub.cfg and fstab: whether every UUID they refer to is present on the server;
• the kernels, their initramfs and modules, and whether the initramfs has the driver for the controller the root lives on;
• the state of the file systems and free space;
• the system's network settings against the server's ports and the address assigned to it.

For Windows: the boot code and active partition or bootmgfw.efi with its firmware entry, where the BCD points, whether Windows starts the driver for its disk controller, the NTFS dirty flag and hibernation (Fast Startup). For the server itself: whether the disks are visible, their SMART health and the state of the software RAID.

Here is the report for a system that was moved to a server with a different RAID controller (an example):

Linux: Debian GNU/Linux 12 (bookworm) on sda2
[ OK ] UEFI loader on the ESP (sda1): grubx64.efi, shimx64.efi.
[ OK ] fstab: all 3 entries are on this server.
[ OK ] Default entry: Debian GNU/Linux
[FAIL] initrd.img-6.1.0-26-amd64 lacks megaraid_sas: the kernel will not
       find the root disk. Rebuild it with the driver (rescue).

Summary: 1 problem(s) that keep it from booting, 0 warning(s)

Without diagnostics, a system like this simply stops during boot with a vague error about the root partition. With it, the cause and the fix are clear right away: go into rescue and rebuild the initramfs with the right driver.

Rescue does all the preparation itself and opens a shell already inside your system. It:

1. Assembles the software RAID, even if the array was created on another machine, and activates LVM.
2. Finds the root file system: ext4, XFS or Btrfs, including in a subvolume.
3. Mounts it together with the other partitions from fstab: /boot and the EFI partition.
4. Binds /dev, /proc, /sys and the UEFI variables so that grub-install and efibootmgr work inside.
5. Opens a shell in the chroot.

From there you work with your system as usual: reinstall GRUB, rebuild the initramfs, edit fstab or reset a password. After exit, Live OS unmounts everything in the right order so the changes are safely written to disk.

If the base Live OS can't handle a file system, it offers to switch to the full one, where Rescue starts on its own on the same console. For Windows, the full Live OS has a separate win-rescue utility. Without starting Windows, it fixes the typical causes of INACCESSIBLE_BOOT_DEVICE, a damaged BCD and stuck updates, and backs up the registry before any change.

Boot the installed OS finds Linux on the disks and starts it straight from Live OS, without rebooting the server. It reads the GRUB configuration, picks the same entry GRUB itself would pick and starts its kernel with the same parameters via kexec. Before starting, the menu shows exactly what will boot: the entry, the kernel, the initramfs and the command line.

This helps in two situations:

• Only the boot loader is broken. The system is intact, but GRUB or the firmware entry is lost. You can bring the server back into service right away and fix the boot loader later, from the running system.
• Maintenance is done. Instead of a reboot with a long hardware check, which takes minutes on servers, the system starts immediately.

The full Live OS returns to the base one the same way, with the base command. Windows can't be started like this: only the firmware can start it. If there is Windows on the disks, Live OS will say so and offer a reboot.

Server configuration shows the server's configuration on one screen. Memory is described the way it is ordered: “32 GB · 2x 16 GB DDR4 2400”. For each network port you get its MAC address, for controllers the model and the configured RAID volumes. On a virtual server it's shorter: cores, amount of memory and disks.

Disks and health covers every disk, including the ones hidden behind a Dell PERC or HP Smart Array hardware RAID controller. Normally this takes vendor utilities; here it doesn't. For each disk you see the model, capacity, type (HDD, SSD, NVMe), temperature and two numbers that make sense without decoding SMART:

• health in percent: “health 93%” instead of raw wear attributes;
• power-on time in years and days: “4y 119d” instead of 37,896 hours.

A worn-out SSD and a disk that reports a problem itself are flagged right away. Diagnostics take the same data into account: sometimes a system won't boot simply because a disk is dying.

You can reinstall the system without leaving the server console. Install an operating system shows a QR code right in the menu window:

1. You scan the code with your phone, and the installation dialog opens in the panel for this exact server.
2. You choose the system and its settings.
3. Live OS picks up the task within a few seconds and runs it right away.

If you are connected over SSH, there is a clickable link next to the QR code. Our engineers use the same QR code approach to run hardware inventory on new servers.

The menu doesn't lock anything away. The S key opens a shell, and its header lists the Live OS commands right away. The same header greets you when you log in over SSH, in both the base and the full Live OS:

  menu        the console menu
  live        switch to the full Live OS
  local-boot  boot the installed OS
  diagnose    check why the installed OS does not boot
  rescue      mount the installed OS and chroot into it to repair it
  hardware    server configuration; hardware disks: disk health

Every menu item is an ordinary command. You can run it by hand, for example pass rescue the partition you need, or see what local-boot -n would boot without starting anything.

The full Live OS gives you all of Debian: a missing tool is installed with apt straight into memory. Unlike “for every occasion” images, you don't have to download gigabytes of utilities in advance: you install only what you need right now.