If you are physically present when an attack is happening, your first response should be to remove the machine from the network by unplugging the network card (if this will not adversely affect any business transactions). Disabling the network at layer 1 is the only true way to keep the attacker out of the compromised box (Phillip Hofmeister's wise advice).
However, some rootkits or back doors are able to detect this event and react to it. Seeing a rm -rf / executed when you unplug the network from the system is not really much fun. If you are unwilling to take the risk, and you are sure that the system is compromised, you should unplug the power cable (all of them if more than one) and cross your fingers. This may be extreme but, in fact, will avoid any logic-bomb that the intruder might have programmed. In this case, the compromised system should not be re-booted. Either the hard disks should be moved to another system for analysis, or you should use other media (a CD-ROM) to boot the system and analyze it. You should not use Debian's rescue disks to boot the system, but you can use the shell provided by the installation disks (remember, Alt+F2 will take you to it) to analyze the system. [46]
The most recommended method for recovering a compromised system is to use a
live-filesystem on CDROM with all the tools (and kernel modules) you might need
to access the compromised system. You can use the mkinitrd-cd
package to build such a CDROM[47]. You might find the FIRE
(previously called Biatchux)
CDROM useful here too, since it's also a live CDROM with forensic tools useful
in these situations. There is not (yet) a Debian-based tool such as this, nor
an easy way to build the CDROM using your own selection of Debian packages and
mkinitrd-cd
(so you'll have to read the documentation provided
with it to make your own CDROMs).
If you really want to fix the compromise quickly, you should remove the
compromised host from your network and re-install the operating system from
scratch. Of course, this may not be effective because you will not learn how
the intruder got root in the first place. For that case, you must check
everything: firewall, file integrity, log host, log files and so on. For more
information on what to do following a break-in, see Sans' Incident Handling
Guide
or CERT's Steps for
Recovering from a UNIX or NT System Compromise
.
Some common questions on how to handle a compromised Debian GNU/Linux system are also available in My system is vulnerable! (Are you sure?), Section 11.2.
Remember that if you are sure the system has been compromised you cannot trust the installed software or any information that it gives back to you. Applications might have been Trojanized, kernel modules might be installed, etc.
The best thing to do is a complete file system backup copy (using
dd
) after booting from a safe medium. Debian GNU/Linux CDROMs can
be handy for this since they provide a shell in console 2 when the installation
is started (jump to it using Alt+2 and pressing Enter). From this shell,
backup the information to another host if possible (maybe a network file server
through NFS/FTP). Then any analysis of the compromise or re-installation can
be performed while the affected system is offline.
If you are sure that the only compromise is a Trojan kernel module, you can try to run the kernel image from the Debian CDROM in rescue mode. Make sure to startup in single user mode, so no other Trojan processes run after the kernel.
The CERT (Computer and Emergency Response Team) is an organisation that can help you recover from a system compromise. There are CERTs worldwide [48] and you should contact your local CERT in the event of a security incident which has lead to a system compromise. The people at your local CERT can help you recover from it.
Providing your local CERT (or the CERT coordination center) with information on
the compromise even if you do not seek assistance can also help others since
the aggregate information of reported incidents is used in order to determine
if a given vulnerability is in wide spread use, if their is a new worm aloft,
which new attack tools are being used. This information is used in order to
provide the Internet community with information on the current security incidents
activity
, and to publish incident notes
and even
advisories
. For
more detailed information read on how (and why) to report an incident read
CERT's
Incident Reporting Guidelines
.
You can also use less formal mechanisms if you need help for recovering from a
compromise or want to discuss incident information. This includes the incidents mailing
list
and the Intrusions mailing
list
.
If you wish to gather more information, the tct
(The Coroner's
Toolkit from Dan Farmer and Wietse Venema) package contains utilities which
perform a 'post mortem' of a system. tct
allows the user to
collect information about deleted files, running processes and more. See the
included documentation for more information.
Some other tools that can be used for forensic analysis provided in the Debian distribution are:
Fenris
.
Strace
.
Ltrace
.
Any of these packages can be used to analyze rogue binaries (such as back
doors), in order to determine how they work and what they do to the system.
Some other common tools include ldd
(in libc6
),
strings
and objdump
(both in binutils
).
If you try to do forensic analysis with back doors or suspected binaries
retrieved from compromised systems, you should do so in a secure environment
(for example in a bochs
or flex86
image or a
chroot
'ed environment using a user with low privileges).
Otherwise your own system can be back doored/r00ted too!
Also, remember that forensics analysis should be done always on the backup copy of the data, never on the data itself, in case the data is altered during analysis and the evidence is lost.
FIXME: This paragraph will hopefully provide more information about forensics in a Debian system in the coming future.
FIXME: talk on how to do a debsums on a stable system with the MD5sums on CD and with the recovered file system restored on a separate partition.
FIXME add pointers to forensic analysis papers (like the Honeynet's reverse
challenge or David
Dittirch's papers
.
Securing Debian Manual
2.99 18 April 2004Wed, 3 Mar 2004 09:18:54 +0100jfs@computer.org