Even the best system administrators have to deal with accidents and unplanned events. Fortunately, the Linode Manager has a number of tools to assist you in the unlikely event that catastrophe strikes your VPS. This guide shows you how to use the tools at your disposal. You can boot your Linode into rescue mode to perform system recovery tasks and transfer data off your disk images, if necessary. And if all else fails, you can rebuild your Linode from a backup or start over with a fresh Linux distribution.
If you suspect that your primary filsystem is corrupted, use the Linode Manager to boot your Linode into rescue mode. This is a safe environment for performing many system recovery and disk management tasks. Rescue mode is based on the Finnix recovery distribution, a self-contained and bootable Linux distribution. You can also use rescue mode for tasks other than disaster recovery, such as formatting disk images to use different filesystems, copying data between disk images, and downloading files from a disk image via SSH and SFTP.
Here's how to boot your Linode into rescue mode:
Make a note of where the disk images are located. For example, in the screenshot shown above, the Ubuntu disk image is at /dev/xvda. You will need this information later.
Your Linode has been booted into rescue mode. To access it, follow the instructions in the next section.
To access your Linode when it's running in rescue mode, you'll need to use the Linode shell (Lish). For instructions, see Console Access and Using the Linode Shell (Lish). Once you've successfully logged in, you'll see the window shown below.
Now you've connected to your Linode, you can start doing stuff in rescue mode.
You can use the fsck system utility (short for "file system check") to check the consistency of file systems and repair any damage detected. If you suspect that your Linode's file system is corrupted, you should run fsck to check for and repair any damage. Here's how:
You should never run fsck on a mounted disk. Do not continue unless you're sure that the target disk image is unmounted.
Run fsck by entering the following command, replacing /dev/xvda with the location of the disk image you want to check and repair:
e2fsck -f /dev/xvda
If no problems are detected, fsck will display a message indicating that the file system is "clean," as shown below.
Once the file system check completes, any problems detected should be fixed. You can try restarting the Linode now. With any luck, fsck fixed your problem and the Linode will boot normally.
By default, your disk images are not mounted when your Linode boots into rescue mode. However, you can manually mount a disk while your Linode is running in rescue mode to perform system recovery and maintenance tasks. Enter the following command to mount a disk in rescue mode, replacing /dev/xvda with the location of the disk image you want to mount:
mount -o barrier=0 /dev/xvda
Disks that contain a single ext3 file system will have mount points under /media in the rescue environment's /etc/fstab file. To view the directories on the disk, enter the following command:
Now you can read and write to files on the mounted disk.
Change root is the process of changing your working root directory. When you change root (chroot) to your Linode root disk image, you will be able to run commands as though you are logged into that system.
Chroot will allow you to change user passwords, remove/install packages, and do other system maintanance and recovery tasks.
Before you can use chroot, you need to mount your root disk image with execute permissions:
mount -o exec,barrier=0 /dev/xvda
Then to create the chroot, you need to mount the temporary filesystems:
cd /media/xvda mount -t proc proc proc/ mount -t sysfs sys sys/ mount -o bind /dev dev/ mount -t devpts pts dev/pts/
Chroot to your disk image with the following command:
chroot /media/xvda /bin/bash
To exit the chroot and get back to Finnix type "exit"
The Finnix recovery distribution does not automatically start an SSH server, but you can start one manually. This is useful if your Linode won't boot and you need to copy files off of the disks. You can also copy entire disk images over SSH. Here's how to start SSH:
Set the root password by entering the following command:
Enter the password for the root user.
Start the SSH server by entering the following command:
Now you can access mounted disks with an SFTP client by using the root user and the password you just set. For instructions on connecting with an SFTP client, see the File Transfer reference manuals. For instructions on copying an entire disk image over SSH, see Copy a Disk Image Over SSH.
The Finnix recovery distribution is based on Debian, so you can use the apt package management system to install additional software packages in the temporary rescue environment. For example, you could install and run the htop utility by issuing the following commands:
apt-get update apt-get install htop htop
The software packages you install will be available as long as your Linode is running in rescue mode.
If you can't rescue an existing disk image, it's time to rebuild your Linode. There are a couple different ways you can do this. You restore from an existing backup and return your Linode to a previous state. If you don't have backups, you can copy files off an existing disk image, erase everything, and start over again from scratch.
If you previously enabled the Linode Backup Service, you may be able to restore one of the backups to your Linode. For instructions on restoring from a backup created by the Linode Backup Service, see Restoring from a Backup. If you created backups with an application other than the Linode Backup Service, review the application's instructions to restore a backup to your Linode.
Did an unauthorized intruder gain access to your Linode? Since it is virtually impossible to determine the full scope of an attacker's reach into a compromised system, you should never continue using a compromised Linode. We recommend that you follow the instructions in Recovering from a System Compromise. You'll need to create a new Linode, copy your existing data from the old Linode to the new one, and then swap IP addresses.
Sometimes it's just easier to erase all of your Linode's disk images and start over with a fresh Linux distribution. This is the "nuclear option" that will erase everything on your Linode.
If you'd like to deploy a new Linux distribution without erasing your existing disk images, see Creating a Disk Image with a Linux Distribution Installed. This is a better option for those who need create a new distribution, but also need to save their existing data.
Here's how to rebuild your Linode from scratch:
Your Linode will boot with the newly-installed Linux distribution. See the Getting Started guide to reprovision your server.
This guide is licensed under a Creative Commons Attribution-NoDerivs 3.0 United States License.
Last edited by Dolores Portalatin on Friday, March 29th, 2013 (r3368).