Sometimes, your system gets compromised. Whether it was because of a weak user password, a vulnerable system package, or an unpatched web application, sometimes you've got to put the brakes on your server and see what you can salvage. Unfortunately, after a hack, it's virtually impossible to determine the full scope of an attacker's reach into a compromised system. The server should not be trusted for production use.
This guide offers a few methods to recover from a system compromise:
If you choose to keep your data, you will need to audit all the content on the affected system to prevent the transfer of malicious components to a new system. Please understand that compromised servers have a negative affect on our network resources, and prompt actions must be taken to prevent service interruption.
This is the easiest option, yet the most destructive. It will wipe all of your data from your Linode and let you start over with a fresh disk.
After following this process, all of your old data and server configurations will be permanently deleted!
This will delete your current compromised images and deploy fresh disks. All data that was stored on the Linode previously will be unrecoverable, but your system will be free of compromise. At this point, we recommend that you follow the instructions in the Securing Your Server guide to disable root logins via SSH, or to disable password logins, in favor of SSH keys only, for all accounts.
If there is data on the compromised Linode that you need to retain, you can use the Finnix rescue environment to examine your old disk images first. Once you have verified the integrity of your data, copy it to the appropriate location on your new server or another offsite location. Our SSH disk copy guide explains how to copy your entire disk image offsite.
You can use a second Linode for the most seamless transition to a new system.
Begin by adding a new Linode to your account. See the Getting Started guide for instructions.
Set a strong password for root and all user accounts, making sure not to reuse any passwords from the compromised system.
Upgrade all system packages:
apt-get update apt-get upgrade --show-upgraded
Follow the instructions in the Securing Your Server guide to disable root logins via SSH, or to disable password logins, in favor of SSH keys only, for all accounts.
Rebuild your production server's configuration on the new Linode.
The next task is to copy your data to the new Linode, and make sure you've purged all compromised portions. Follow these steps:
Create a temporary directory on the new Linode.
Copy any needed user and configuration data from the compromised Linode using rsync or scp. If you are not familiar with these programs, you can find more information by entering the man rsync or man scp commands.
Do not log in to the new Linode from the compromised Linode. Files should be pulled from the compromised server to your new setup instead.
Alternately, if you're not comfortable copying anything from the compromised system to your new server prior to auditing the data, you may wish to use the Finnix rescue environment to examine your old disk images first. Once you have verified the integrity of your data, copy it to the appropriate location on your new server.
Next, you'll want to swap IP addresses so the new Linode uses the IP address assigned to the old Linode. Please note that if you configured any network services to use the new Linode's IP address, you will most likely want to modify their configurations now to use the old Linode's IP instead. For instructions, see Swapping IP Addresses.
To swap IP addresses, both Linodes must be located in the same data center.
Alternately, you may wish to update your DNS entries to point to the new Linode's IP address instead. Please be aware that DNS propagation across the Internet may take some time. Boot the new Linode to resume normal operations.
You may wish to download a complete copy of the old Linode's disk image(s) for forensic analysis. To do this, follow the instructions in our SSH disk copy guide. Even if you don't need a full copy of the affected disk images, you may still want to make a copy of all system log files for later review.
When you no longer need the old Linode's disk images, you should remove the Linode. Your account will be issued a pro-rated credit for that Linode's charges in the current billing period.
This guide is licensed under a Creative Commons Attribution-NoDerivs 3.0 United States License.
Last edited by Sharon Campbell on Friday, July 12th, 2013 (r3532).