Securing Debian Manual
Chapter 10 - After the compromise (incident response)

10.1 General behavior

If you are physically present when an attack is happening, your first response should be to remove the machine from the network by unplugging the network card (if this will not adversely affect any business transactions). Disabling the network at layer 1 is the only true way to keep the attacker out of the compromised box (Phillip Hofmeister's wise advice).

However, some rootkits or back doors are able to detect this event and react to it. Seeing a rm -rf / executed when you unplug the network from the system is not really much fun. If you are unwilling to take the risk, and you are sure that the system is compromised, you should unplug the power cable (all of them if more than one) and cross your fingers. This may be extreme but, in fact, will avoid any logic-bomb that the intruder might have programmed. In this case, the compromised system should not be re-booted. Either the hard disks should be moved to another system for analysis, or you should use other media (a CD-ROM) to boot the system and analyze it. You should not use Debian's rescue disks to boot the system, but you can use the shell provided by the installation disks (remember, Alt+F2 will take you to it) to analyze the system. [46]

The most recommended method for recovering a compromised system is to use a live-filesystem on CDROM with all the tools (and kernel modules) you might need to access the compromised system. You can use the mkinitrd-cd package to build such a CDROM[47]. You might find the FIRE (previously called Biatchux) CDROM useful here too, since it's also a live CDROM with forensic tools useful in these situations. There is not (yet) a Debian-based tool such as this, nor an easy way to build the CDROM using your own selection of Debian packages and mkinitrd-cd (so you'll have to read the documentation provided with it to make your own CDROMs).

If you really want to fix the compromise quickly, you should remove the compromised host from your network and re-install the operating system from scratch. Of course, this may not be effective because you will not learn how the intruder got root in the first place. For that case, you must check everything: firewall, file integrity, log host, log files and so on. For more information on what to do following a break-in, see Sans' Incident Handling Guide or CERT's Steps for Recovering from a UNIX or NT System Compromise.

Some common questions on how to handle a compromised Debian GNU/Linux system are also available in My system is vulnerable! (Are you sure?), Section 11.2.

10.2 Backing up the system

Remember that if you are sure the system has been compromised you cannot trust the installed software or any information that it gives back to you. Applications might have been Trojanized, kernel modules might be installed, etc.

The best thing to do is a complete file system backup copy (using dd) after booting from a safe medium. Debian GNU/Linux CDROMs can be handy for this since they provide a shell in console 2 when the installation is started (jump to it using Alt+2 and pressing Enter). From this shell, backup the information to another host if possible (maybe a network file server through NFS/FTP). Then any analysis of the compromise or re-installation can be performed while the affected system is offline.

If you are sure that the only compromise is a Trojan kernel module, you can try to run the kernel image from the Debian CDROM in rescue mode. Make sure to startup in single user mode, so no other Trojan processes run after the kernel.

10.3 Contact your local CERT

The CERT (Computer and Emergency Response Team) is an organisation that can help you recover from a system compromise. There are CERTs worldwide [48] and you should contact your local CERT in the event of a security incident which has lead to a system compromise. The people at your local CERT can help you recover from it.

Providing your local CERT (or the CERT coordination center) with information on the compromise even if you do not seek assistance can also help others since the aggregate information of reported incidents is used in order to determine if a given vulnerability is in wide spread use, if their is a new worm aloft, which new attack tools are being used. This information is used in order to provide the Internet community with information on the current security incidents activity, and to publish incident notes and even advisories. For more detailed information read on how (and why) to report an incident read CERT's Incident Reporting Guidelines.

You can also use less formal mechanisms if you need help for recovering from a compromise or want to discuss incident information. This includes the incidents mailing list and the Intrusions mailing list.

10.4 Forensic analysis

If you wish to gather more information, the tct (The Coroner's Toolkit from Dan Farmer and Wietse Venema) package contains utilities which perform a 'post mortem' of a system. tct allows the user to collect information about deleted files, running processes and more. See the included documentation for more information.

Some other tools that can be used for forensic analysis provided in the Debian distribution are:

• Fenris.

• Strace.

• Ltrace.

Any of these packages can be used to analyze rogue binaries (such as back doors), in order to determine how they work and what they do to the system. Some other common tools include ldd (in libc6), strings and objdump (both in binutils).

If you try to do forensic analysis with back doors or suspected binaries retrieved from compromised systems, you should do so in a secure environment (for example in a bochs or flex86 image or a chroot'ed environment using a user with low privileges). Otherwise your own system can be back doored/r00ted too!

Also, remember that forensics analysis should be done always on the backup copy of the data, never on the data itself, in case the data is altered during analysis and the evidence is lost.

FIXME: This paragraph will hopefully provide more information about forensics in a Debian system in the coming future.

FIXME: talk on how to do a debsums on a stable system with the MD5sums on CD and with the recovered file system restored on a separate partition.

FIXME add pointers to forensic analysis papers (like the Honeynet's reverse challenge or David Dittirch's papers.

Securing Debian Manual