Chapter 9. Detecting And Managing A Break-in

At a Glance

Despite your best efforts, you may have to deal with a compromised system. This chapter discusses how you can use auditing, logging, and forensics to help detect compromises and identify what’s been modified on a compromised system, and provides step by step guidance for how to recover from an attack.

Auditing and Logging

After you have established the protection mechanisms on your system, you will want to be sure that your protection mechanisms actually work. You will also want to observe any indications of misbehavior or other problems. This process is known as monitoring or auditing.

Two of the most common audits are inspections of file integrity and review of system log files.

File Integrity Checks

Although there are many reasons that you might want to examine the integrity of your system’s files, one of the most common reasons is to determine what has changed after a computer has been attacked, broken into, and compromised.

There are basically three approaches to detecting changes to files:

1. Use comparison copies of the data to be monitored; this is the most certain way.

2. Monitor metadata about the items to be protected; this includes monitoring the modification time of entries as kept by the operating system, and monitoring any logs or audit trails that show alterations to files.

3. Use some form of signature of the data to be monitored, and periodically recompute and compare the signature against a stored value.

Each of these approaches has drawbacks and benefits. Whichever you choose, there are several ways you can examine a potentially compromised system:

• Physically remove the hard disk from the computer in question, attach the disk to a second computer as an auxiliary disk, boot the second computer, mount the disk read-only, and use the second computer’s operating system to examine the disk (or make a block-for-block copy to examine).

• Leave the suspect disk in the suspect computer, but boot the suspect computer with a clean operating system from a CD-ROM or a floppy disk. Then, using only the tools on the CD-ROM or floppy, you could proceed to mount the suspect disk read-only and analyze the possibly compromised filesystem.

• Log into the suspect computer and run whatever integrity-checking tools happen to be installed.

Clearly, the most thorough way to examine the suspect system is the first technique. In practice, the third technique is the most common, but is completely inadequate. If an attacker truly compromises your computer system, nothing can be trusted – including the integrity-checking software or databases.

Comparison Copies

The safest and most direct method of detecting changes to data is to keep a copy of the unaltered data, and do a byte-by-byte comparison when needed. If there is a difference, this indicates not only that a change occurred, but what that change involved.

Comparison copies, however, are unwieldy. They require that you keep copies of every file of interest. Not only does such a method require twice as much storage as the original files, it also may involve a violation of license or copyright of the files (in general, copyright laws allow one copy for archival purposes, and your distribution media is that one copy).243 To use a comparison copy means that both the original and the copy must be read through, byte by byte, each time a check is made. And, of course, the comparison copy needs to be saved in a protected location.

Even with these drawbacks, comparison copies have a particular benefit—if you discover an unauthorized change, you can simply replace the altered version with the saved comparison copy, thus restoring the system to normal. These copies can be made locally, at remote sites, or over the network, as we describe in the following sections.

Local copies

One standard method of storing comparison copies is to put them on another disk, particularly on removable media. Many people report success with storing copies of critical system files on removable media drives.244 If there is any question about a particular file, the appropriate disk is placed in the drive, mounted, and compared. If you are careful about how you configure these disks, you get the added (and valuable) benefit of having a known good version of the system to boot up if the system is compromised by accident or attack. Making regular backups to removable or write-once media such as tapes and CDs can provide similar benefits.

A second standard method of storing comparison copies is to make on-disk copies somewhere else on the system. You can compress and/or encrypt the copy to help reduce disk use and keep it safe from tampering. The disadvantage to compression and encryption is that it then requires extra processing to recover the files if you want to compare them against the working copies. This extra effort may be significant if you wish to do comparisons daily (or more often!). Moreover, you can’t protect the encryption program itself this way.

Remote copies

A third method of using comparison copies is to store them on a remote site and make them available remotely in some manner. For instance, you might place copies of all the system files on a disk partition on a secured server, and export or share that partition read-only using NFS or some similar protocol. All the client hosts could then mount that partition and use the copies in local comparisons. Of course, you need to ensure that whatever programs are used in the comparison are taken from the remote partition and not from the local disk. Otherwise, an attacker could modify those files to not report changes!

Another method of remote comparison involves using a program such as rdist to do the comparison across the network. (PUIS, 626-627) Remember that it is not enough to keep copies of executable programs. Shared libraries and configuration files must usually be compared as well.

Checklists and Metadata

Saving an extra copy of each critical file and performing a byte-by-byte comparison can be unduly expensive. It requires substantial disk space to store the copies. Furthermore, if the comparison is performed over the network, it will involve substantial disk and network overhead each time the comparisons are made.

A more efficient approach is to store a summary of important characteristics of each file and directory. When the time comes to do a comparison, the characteristics are regenerated and compared with the saved information. If

243 Copyright laws—and many licenses—do not allow for copies on backups.

244 Note that an external Firewire-based disk drive fits this description.

the characteristics are comprehensive and smaller than the file contents (on average), then this method is clearly a more efficient way of doing the comparison.

Furthermore, this approach can capture changes that a simple comparison copy cannot: comparison copies detect changes in the contents of files, but do little to detect changes in metadata such as file owners, protection modes, or modification times. These data are sometimes more important than the data within the files themselves. For instance, changes in owner or protection bits may result in disaster if they occur to the wrong file or directory. The simplest form of a checklist mechanism is to list the files with their attributes, and compare the output against a saved version of the list. It’s usually necessary to include all the ancestor directories of important files in the checklist as well.

Checksums and Signatures

Unfortunately, simple checklists can be defeated with a little effort. Files can be modified in such a way that the information you monitor will not disclose the change. For instance, a file might be modified by writing to the raw disk device after the appropriate block is known. As the modification did not go through the filesystem, none of the information about file change time will be altered. Or an attacker could set the system clock back to the time of the last legitimate change, edit the file, and set the clock forward again.

To protect against these threats, we can generate a signature for each file, and compare file signatures. A good file signature must depend on every bit in the file, and it should be infeasible for an attacker to create another file that produces the same signature. These requirements disqualify simple checksum algorithms (like CRC), but are met by cryptographic message digests (discussed in chapter 5-3).

Well-developed software for file integrity checking computes at least one, and often several cryptographic digests of each file and its metadata. When a known good copy of the checker is used to generate file signatures in advance, and these signatures are stored safely (on write-once or removable media, for example) any changes to the files can be detected by running the known good copy again and comparing the signatures. One well known multi-platform package for doing this is Tripwire (http://www.tripwire.com); an open source version is available for Linux at no cost.

Log Files

A log file is a file that records one or more log events — that is, a specific action, activity or condition that the author of a program thought might be worth recording. Log files are an important building block of a secure system: they form a recorded history, or audit trail, of your computer’s past, making it easier for you to track down intermittent problems or attacks. Using log files, you may be able to piece together enough information to discover the cause of a bug, the source of a break-in, or the scope of the damage involved. In cases where you can’t stop damage from occurring, at least you will have some record of it. Those logs may be exactly what you need to rebuild your system, conduct an investigation, give testimony, recover insurance money, or get accurate field service performed.

Logs can be recorded in multiple locations:

• The logs can be stored on the computer responsible for the log event. For example, on modern Unix systems, logs are stored in the directory /var/log, although other directories can be used by specific programs in specific cases. Windows NT-based systems collect messages from the operating system and applications about events and store them in a unified log file (often C:\WINNT\system32\config\SysEvent.Evt), although individual applications may also maintain their own log files.

• The logs can be sent over the network to a remote computer to be aggregated and stored. This computer, sometimes called a log server, can be used as a central location for monitoring many computers on a network. A log server can further be configured with a host-based firewall so that it can receive log information from other computers, but so that the computer is prohibited from transmitting any packets on the network. Using a remote log server helps prevent attackers from erasing their tracks. A centralized, remote logging system may also be an ideal place to run intrusion detection software on the collected logs.

• The logs can be written to write-once media or printed on a printer. These logs are virtually impossible to erase without physical access, but can become unwieldy to maintain.

For security reasons, some information should never be logged. For example, although you should log failed password attempts, you should not log the password that was used in the failed attempt. Users frequently mistype their own passwords, and logging these mistyped passwords would help an attacker break into a user’s account. Some system administrators believe that the account name should also not be logged on failed login attempts— especially when the account typed by the user is nonexistent. The reason is that users occasionally type their passwords when they are prompted for their usernames.

Essential Log Events

Although different systems and applications log different events, some kinds of events are essential, and should be logged by any reasonably secure computer:

• Network connections from remote hosts, dialup connections on modems, and dial-out connections by modems. In some cases, logging overall data traffic patterns may reveal excessive outgoing data caused by an attacker using the computer as a staging ground for pirated software, or publishing your confidential data.

• User login times and locations. Seeing someone logging into the account of a local user from out of the country or at unusual hours may signal an intruder.

• Failed login attempts, which may alert you to attackers knocking on your computer’s door.

• Process-level accounting, including process start and end times, ownership and privileges, and CPU utilization. This kind of accounting can reveal every command issued on your system, and is very helpful in analyzing security breaches – if it’s intact.

• System shutdowns and reboots. Unexpected reboots may indicate a hardware problem, an attacker with physical access restarting the system in single-user mode, or a remote attacker covering his tracks in memory.

• Exceptional events reported by the operating system (such as disk full conditions). These always require attention, whether or not they are caused by an attacker.

Every event that’s logged should include the process that generated the event and the date and time that it occurred. Most logging systems assign each logged event to a category or facility that describes the source of the event (such as ‘mail’ or ‘network’ or ‘kernel’), and a priority or severity that describes the importance of the event (such as ‘informational’, ‘warnings’, or ‘critical error’). Here is an example of a message logged by a Unix system:

Aug 14 08:02:12 <mail.info> r2 postfix/local[81859]: 80AD8E44308: to=<jhalonen@ex.com>, relay=local, delay=1, status=bounced (unknown user: "jhalonen")

This message was generated by the postfix program called local. It reports that an e-mail message with the id 80AD8E44308 was received for the user jhalonen@ex.com. The message was bounced, because there is no user jhalonen@ex.com. The event’s facility is mail; the priority is info. (See PUIS, 642-654 for a detailed discussion of Unix logging facilities, priorities, and configuration.)

Log File Analysis

It’s not enough to log events – you must read the logs. On a busy server that may log hundreds or thousands of events an hour, this can be a daunting task. Log file analysis programs attempt to streamline the job by filtering log files to direct your attention toward important events and away from routine ones.

Some analysis software, like Microsoft’s Event Viewer, lets you interactively view logs through filters of your choice. Other software, like the Swatch program often used on Unix servers, monitors logs in real-time and issues alerts when important events occur. (PUIS, 654-657)

Program-Specific Log Files

Most application programs, especially daemons, will maintain their own log files. FTP and web servers routinely log connections and file transfers, DNS name servers log domain transfers and queries, database servers log queries, and mail servers routinely log connection and message size information when sending or receiving messages. Errors and exceptional conditions are nearly always logged. In many cases, application-specific log analysis tools have been developed to summarize these logs in a more useful fashion.

Handwritten Logs

Another type of logging that can help you with security is not done by the computer at all; it is done by you and your staff. Keep a log book that records your day’s activities. Log books should be kept on paper in a physically secure location. Because you keep them on paper, they cannot be altered by someone hacking into your computer even as superuser. They will provide a nearly tamperproof record of important information.

Handwritten logs have several advantages over online logs. You can record things that the computer won’t, like bomb threats. You can access paper logs when the system is down. In some countries, there can be legal advantages to paper logs as evidence.

The biggest problem with log books is the amount of time you need to keep them up to date. These are not items that can be automated with a shell script. Unfortunately, this time requirement is the biggest reason why many administrators are reluctant to keep logs—especially at a site with hundreds (or thousands) of machines, each of which might require its own log book. We suggest that you try to be creative and think of some way to balance the need for good records against the drudgery of keeping multiple books up to date. Compressing information, and keeping logs for each cluster of machines are ways to reduce the overhead while receiving (nearly) the same benefit.

There are basically two kinds of log books: per-site logs and per-machine logs. In a per-site log book, you want to keep information that would be of use across all your machines and throughout your operations. The information can be further divided into exception and activity reports (power outages, alarm tests and triggers, personnel actions on employees with privileged access), and informational material (contact information, receipts for hardware and software, equipment serial numbers, MAC addresses for Ethernet machines, copies of router configurations, etc.). Each machine should also have a log book associated with it. In a per-machine log, exception and activity reports include notes about system crashes, downtimes, account creation and deletion, password changing, software installation, and system backups. Informational material might include copies of configuration files, lists of patches applied, and disk configurations.

Managing Log Files

Here are several final suggestions about log files:

Backups

Ensure that all of your log files are copied to your backup media on a regular basis, preferably daily. The timing of the backups should be such that any file that is periodically reset is copied to the backups before the reset is performed. This will ensure that you have a series of records over time to show system access and behavior.

Review

Review log files at least daily. Keeping log records does you little service if you do not review them on a regular basis. Log files can reveal problems with your hardware, with your network configuration, and (of course) with your security.

Processing

Filter your log files with analysis software. Many log messages record nothing of particular interest. You may become so accustomed to seeing this material that you get in the habit of making only a cursory scan of the messages to see if something is wrong, and this way you can easily miss an important message.

Filtering requires some care. You do not want to write a filter that selects those important things you want to see and discards the rest. Such a system is likely to result in an important message being discarded without being read. Instead, you should filter out the boring messages, being as specific as you can with your pattern matching, and pass everything else to you to be read. Periodically, you should also study unfiltered log messages to be sure that you are not missing anything of importance.

Trust

Don’t trust your logs completely! Logs can often be altered or deleted by an intruder who obtains superuser privileges. Local users with physical access or appropriate knowledge of the system may be able to falsify or circumvent logging mechanisms. And, of course, software errors and system errors may result in logs not being properly collected and saved. Thus, you need to develop redundant scanning and logging mechanisms: because something is not logged does not mean it didn’t happen.

Of course, simply because something was logged doesn’t mean it did happen, either—someone may cause entries to be written to logs to throw you off the scent of a real problem or to point a false accusation at someone else.

Forensics

The information in log files is, for the most part, intentionally put there as a result of a programmer’s decision. But a running system records other information as well. In recent years, there has been significant interest in computer forensics, the art of reading the tracks that are left in a computer system.

Although not obvious, some files are often kept on a per-user basis can be helpful in analyzing when something untoward has happened on your system. While not real log files, as such, these files can be treated as possible sources of information on user behavior.

Shell History

Many of the standard Unix user command shells, including bash, csh, tcsh, and ksh, can keep a history file. When the user issues commands, the text of each command and its arguments are stored into the history file for later reexecution. If you are trying to recreate activity performed on an account, possibly by some intruder, the contents of this file can be quite helpful when coupled with system log information. You must check the modification time on the file to be sure that it was in use during the time the suspicious activity occurred. If it was created and modified during the intruder’s activities, you should be able to determine the commands run, the programs compiled, and sometimes even the names of remote accounts or machines that might also be involved in the incident. Be sure of your target, however, because this is potentially a violation of privacy for the real user of this account.

Obviously, an aware intruder will delete the file before logging out. In some cases, however, you can preserve the file either by forcing the intruder to log out, by making a hard link to the file elsewhere before the intruder logs out, or by recovering the deleted file. (PUIS, 677-678)

Mail

Some user accounts are configured to make a copy of all outgoing mail in a file. If an intruder sends mail from a user account where this feature is set, the message copies can provide you with potentially useful information. In at least one case we know of, a person stealing confidential information by using a coworker’s pirated password was exposed because of recorded e-mail to his colleagues that he signed with his own name!

Network Setup

Each user account may have several network configuration properties or files that can be edited to provide shortcuts for commands, or to assert access rights. Sometimes, the information in these files will provide a clue as to the activities of a malefactor. Unix examples include the .rhosts, .ssh/known_hosts, and .ssh/authorized_keys files for remote logins, and the .netrc file for FTP. Examine these files carefully for clues, but remember: the presence of information in one of these files may have been there prior to the incident, or it may have been planted to throw you off.

Handling a Break-in

You should have an action plan prepared to deal with a security breach. Particularly security-conscious organizations should practice the plan. Here are the key components you should include:

Step 1: Identify and understand the problem.

Don’t panic, and don’t act without thinking. If you don’t know what the problem is, you cannot take action against it. This rule does not mean that you need to have perfect understanding, but you should understand at least what form of problem you are dealing with. Cutting your organization’s Internet connection won’t help you if the problem is being caused by a revenge-bent employee with a laptop who is hiding out in a co-worker’s office.

Step 2: Document.

Whether your goal is to get your system running again as soon as possible, or to collect evidence for a prosecution, you will be better off if you document what you do. Start a paper log immediately. Take a notebook and write down everything you find, noting the date and time. If you examine text files, print copies, and then sign and date the hardcopy.

In larger organizations, there may be an internal response team or security officer tasked to handle break-ins, contain damage, or coordinate responses. If your organization has such an internal incident reporting system, insure that they are involved at an early stage to assist with preservation of documentation as well as response.

Step 3: Contain or stop the damage.

If you’ve identified the problem, take immediate steps to halt or limit it. For instance, if you’ve identified the employee who is deleting system files, you’ll want to turn off his account, and probably take disciplinary action as well. Both are steps to limit the damage to your data and system.

Step 4: Confirm your diagnosis and determine the damage.

After you’ve taken steps to contain the damage, confirm your diagnosis of the problem and determine the damage it caused. Are files still disappearing after the employee has been discharged? You may never be 100% sure if two or more incidents are actually related. Furthermore, you may not be able to identify all of the damage immediately, if ever.

Step 5: Preserve the evidence, if necessary.

If you intend to prosecute or seek legal redress for your incident, you must make an effort to preserve necessary evidence before going further. Failure to preserve evidence does not prohibit you from calling the police or filing a suit against the suspected perpetrator, but the lack of evidence may significantly decrease your chances for success. Be advised: preserving evidence can take time and is hard to do properly. For this reason, many organizations dealing with incidents forgo this step.

Step 6: Restore your system.

After you know the extent of the damage, you need to restore the system and data to a consistent state. This may involve reloading portions of the system from back-ups, or it may mean a simple restart of the system. Before you proceed, be certain that all of the programs you are going to use are “safe.” The attacker may have replaced your restore program with a Trojan horse that deletes both the files on your hard disk and on your backup tape!

Step 7: Deal with the cause.

If the problem occurred because of some weakness in your security or operational measures, you’ll want to make changes and repairs after your system has been restored to a normal state. If the cause was a person making a mistake, you will probably want to educate him or her to avoid a second occurrence of the situation. If someone purposefully interfered with your operations, you may wish to involve law enforcement authorities.

Step 8: Perform related recovery.

If damage caused by the attack is covered by insurance, you may need to file claims. Rumor control, and perhaps even community relations, will be required at the end of the incident to explain what happened, what breaches occurred, and what measures were taken to resolve the situation. This step is especially important with a large user community, because unchecked rumors and fears can often damage your operations more than the problem itself.

Step 9: Postmortem.

Once the heat has died down, review the incident and your handling of it. How could you and your team have handled the situation better? What effort was wasted? What wrong decisions were made? How could you have prevented it from happening in the first place?

In addition to having a plan of action, you can be prepared by creating a toolkit on read-only media (floppy, CD, etc.) This toolkit will give you a set of programs for incident response that you know are not compromised. Include programs that you will need to examine a compromised system. For a Unix system, these might include: awk, bash, cat, compress, cut, dd, des, df, du, file, find, grep, gzip, icat, ifconfig, last, ls, lsmod, lsof, md5sum, modinfo, more,

netcat, netstat, nmap, paste, pcat, perl, pgp, pkginfo, ps, rpm, rm, script, sed, strings, strace, tar, top, truss, uncompress, vi, and w. Don’t forget shared libraries (or insure that the programs are statically linked). Having a bootable live filesystem on your CD or DVD is useful as well. One particularly handy toolkit is Knoppix

(http://www.knoppix.org), a bootable live Linux CD that includes a myriad of analysis and forensics tools. Because Linux can mount Microsoft FAT file systems and other Unix file systems, a Knoppix CD makes an excellent general forensic toolkit.

Discovering an Intruder

There are several ways you might discover a break-in:

• Catching the perpetrator in the act. For example, you might see the superuser logged in from a cyber-cafe in Budapest when you are the only person who should know the superuser password.

• Deducing that a break-in has taken place based on changes that have been made to the system. For example, you might receive an electronic mail message from an attacker taunting you about a security hole, you may discover new accounts, or your network connection might be running very slowly because the bandwidth is being used by people downloading copyrighted software.

• a message from a system administrator at another site indicating strange activity at his or her site that has originated from an account on your machine.

• Strange activities on the system, such as system crashes, significant hard disk activity, unexplained reboots, minor accounting discrepancies, or sluggish response when it is not expected.

There are a variety of programs that you can use to check files and processes that might lead you to discover a break-in. Use these tools on a regular basis, but use them sporadically as well. This introduces an element of randomness that can keep perpetrators from being able to cover their tracks. This principle is a standard of operations security: try to be unpredictable.

What to Do When You Catch Somebody

You have a number of choices when you discover an intruder on your system:

1. Ignore them — they might go away. This is generally a poor choice. Ignoring an intruder who is on your system essentially gives him free reign to do harm to you, your users, and others on the network. You may also put yourself at risk for downstream liability if the intruder causes damages at another organization and you had the chance to stop him.

2. Try to contact them, and ask them what they want. Be extremely careful if you pursue this course of action. Some intruders are malicious in intent, or extremely paranoid about being caught. If you contact them, they may react by trying to delete everything on your computer so as to hide their activities. Trace the intruder before you contact them, and document every contact.

3. Monitor the intruder. This will give you an idea of whether the intruder is modifying your accounting database or simply rummaging around through your users’ e-mail. However, keep in mind that you don’t know how long this intruder has been on your system, so all you can really monitor is what is done next.

If the intruder is logged in over a network connection, you can use a packet monitor such as tcpdump, ethereal, or snoop to either display the user’s packets or record them in a file. If your computer is attached to a hub, another computer on the same network may be able to capture packets unobtrusively.

If your intruder is logged on through a modem or serial port that is connected directly to your computer, there are several programs that you can use to monitor the intruder’s actions, including ttywatch, conserver, rtty, and ser2net. These programs can give you a detailed, byte-by-byte account of the information that is sent over one

or more serial ports. In many cases, they can also monitor pseudo-ttys, which is valuable when the attacker has connected over the network using an encrypted protocol such as SSH.

In some countries, monitoring an intruder may be illegal, or may only be legal if you have a banner telling all users that they may be monitored.

4. Try to trace the connection and identify the intruder. If the intruder has called your computer by telephone, your options will depend on the services offered by your telephone company; some offer a caller id service or caller tracing services. If the intruder has logged in over the network, the who or netstat commands may quickly identify the computer that originated the connection. You can then contact the system administrator of the remote machine (by telephone!) for further tracing; administrator contact information can often be found in the whois registry for their domain name, or on the organization’s web site. Another option is to use a traceroute program (traceroute or tracert.exe) to identify the network provider for the remote machine. If all else fails, you might send e-mail to root or postmaster at the remote machine asking them to call you; don’t mention the break-in, however, as the intruder may be monitoring that account.

5. Break their connection by killing their processes, unplugging the modem or network, or turning off your computer. Killing your computer’s power—turning it off—is the very quickest way to get an intruder off your computer and prevent him from doing anything else—including possibly further damage. Unfortunately, this is a drastic action. Not only does it stop the intruder, but it also interrupts the work of all of your legitimate users. It may also delete evidence you night need in court some day, delete necessary evidence of the break-in, such as running processes, and cause the system to be damaged when you reboot because of Trojaned startup scripts. In addition, many file systems do not deal with sudden power loss very gracefully: pulling the plug might do significantly more damage than the intruder might ever do.

On Unix systems, you can use the ps command to get a list of the intruder’s processes and the kill command to stop them, after you change the password on the account that the intruder is using. On Windows systems, the Task Manager serves these functions.

If the intruder is connecting over a network, you can break that network connection by programming your firewall or router to drop packets from the user’s host, or unplug the network connector altogether; if the intruder has dialed in over a telephone line, you can turn off the modem—or unplug it from the back of the computer.

6. Contact your Internet Service Provider, an incident response team, or law enforcement official to notify them of the attack.

After the Attack

The remainder of this chapter discusses in detail how to find out what an intruder may have done and how you should clean up afterwards.

Analyzing the Log Files

Even if you don’t catch an intruder in the act, you still have a good chance of finding the intruder’s tracks by routinely looking through the system logs. Look for things out of the ordinary; for example:

• Users logging in at strange hours

• Unexplained reboots

• Unexplained changes to the system clock

• Unusual error messages from the mailer, ftp daemon, or other network server

• Failed login attempts with bad passwords

• Unauthorized or suspicious use of the su command

• Users logging in from unfamiliar sites on the network

On the other hand, if the intruder is sufficiently skillful and achieves superuser access on your machine, he or she may erase all evidence of the invasion. Simply because your system has no record of an intrusion in the log files, you can’t assume that your system hasn’t been attacked.

Many intruders operate with little finesse: instead of carefully editing out a record of their attacks, they simply delete or corrupt the entire log file. This means that if you discover a log file deleted or containing corrupted information, there is a possibility that the computer has been successfully broken into. However, a break-in is not the only possible conclusion. Missing or corrupted logs might mean that one of your system administrators was careless; there might even be an automatic program in your system that erases the log files at periodic intervals.

You may also discover that your system has been attacked if you notice unauthorized changes in system programs or in an individual user’s files. This is another good reason for using a file integrity tool to monitor your files for changes.

If your system logs to a hardcopy terminal or another computer, you may wish to examine that log first, because you know that it can’t have been surreptitiously modified by an attacker coming in by the telephone or network.

Preserving the Evidence

If you wish to prosecute your attacker (if you ever find the person) or sue them for damages, you will need to preserve some evidence that a crime has been committed. Even if you do not wish to take any legal action, you may find it useful to collect evidence of the attack so that you have the ability to reconstruct what happened.

There are many approaches to gathering evidence. Here are some approaches that we have found successful:

1. Capture the data in the system’s memory. On Unix, you can use the dd command: # dd bs=1024 < /dev/mem > mem.image

# dd bs=1024 < /dev/kmem > kmem.image

2. Make a complete copy of the computer’s disk drives. Now remove the original disks, place them in a vault, and work with the copies on another machine. If your system uses the /proc filesystem, the copied /proc may be of particular interest.

3. Copy key files that were left or modified by the intruder into an archive. Make a copy of this archive on several computers.

4. Write modified files to CDR or DVD-RAM media.

5. Run arp -a or arp -v to print the contents of the ARP table, which may suggest network connections that have been recently established.

6. If your web site was defaced, save the HTML pages on your computer’s hard drive. Use a screen capture utility to record a copy of how the image looked on your computer’s screen.

7. Take copies of displays that might reflect the current state of the compromised system. X-Windows systems can use xwd for this purpose; Microsoft Windows systems can use the PrtSc key.

8. Compute the MD5 digest of any images or files that you recover. Print the MD5 on a piece of paper, sign it, date it, and put it in your incident log book. You can use this MD5 at a later point in time to establish that the evidence has not been altered.

There are commercial products that you may find useful to assist you in preserving evidence, including high-speed disk duplicators and network forensics analysis tools (NFATs) that record all packets entering and leaving an organization.

If you have involved law enforcement authorities, speak with them before attempting to preserve any evidence on your own.

Cleaning Up After the Intruder

If your intruder gained superuser or administrator access, or access to another privileged account such as mail, he may have modified your system to make it easier for him to break in again in the future. If the intruder has installed a password sniffer or stolen the password file, he’ll potentially have access to a legitimate account and will be able to get back in no matter what other precautions are taken. You’ll have to change all of the passwords on the system.

After a successful break-in, you must perform a careful audit to determine the extent of the damage. Depending on the nature of the break-in, you’ll have to examine your entire system. You may also need to examine other systems on your local network, or possibly the entire network (including routers and other network devices).

An intruder can compromise a system in many ways that can be difficult or impossible to detect. The safest course of action is to reinstall the operating system from scratch, apply all security patches, reinstall application programs from scratch (along with their patches) and then carefully restore user files from backups, or, if necessary, the compromised disks. Here are some particularly common things to watch for in your audit:

New Accounts

After a break-in, scan for newly created accounts. Delete any accounts that have been created by an intruder. You may wish to make a paper record of each account before deleting it in case you wish to prosecute the intruder (assuming that you ever find the villain).

Changes in file contents or permissions

An intruder who gains privileges can change any file on your system. Although you should make a thorough inventory of your computer’s entire filesystem, you should look specifically for any changes to the system that affect security. For example, an intruder may have inserted trap doors or logic bombs to do damage at a later point in time. A clean copy of a file integrity checker and a known good backup of its database are invaluable.

New SUID and SGID files

Intruders who gain superuser access frequently create SUID and SGID files when the system supports them. After a break-in, scan your system to make sure that new SUID files have not been created. (PUIS, 151-154)

Changes in network access files

An intruder may have created or changed files to allow remote access in the future. For example, under Unix, the intruder may create new .rhosts or.ssh/authorized_keys files in your users’ home directories, or added machines to the system-wide /etc/hosts.equiv file. Check all of these files and ask users to do the same. (PUIS, 705-706)

Changes to startup files

An intruder may have modified the contents of user or system-wide startup files, or files that may be automatically run at scheduled times or when triggered by certain events (like automatically forwarding an e-mail message). All these files need to be carefully checked.

Change to configuration files

Any service that runs as a privileged user and reads a configuration file may be vulnerable to changes made to that configuration file. The Windows Registry is the epitome of vulnerable configuration files. All configuration files for services should be checked against known good copies or cryptographic signatures.

Hidden files and directories

The intruder may have created “hidden directories” on your computer, and may be using them as a repository for stolen information or for programs that break security.

Intruders often hide their files in directories with names that are a little difficult to discover or enter on the command line. This way, a novice system administrator who discovers the hidden directory will be unlikely to figure out how to access its contents. Filenames that are difficult to discover or enter include “.. ” (dot dot space), and names containing control characters, backspaces, or other special characters. Some names can be entered in Unicode that display as familiar alphabetic characters but that cannot be entered easily from the keyboard. Another approach is to use filenames that sound like they are obscure parts of the operating system that should not be tampered with (file systems that have “system” attributes for directories are particularly vulnerable to this approach.)

Unowned files

Sometimes attackers leave files in the filesystem that are not owned by any user or group. This can happen if the attacker created an account and some files, and then deleted the account—leaving the files. Alternatively, the attacker might have been modifying the raw data on a disk and changed a UID by accident.

New network services

Many intruders (and many attack scripts) will install network daemons that provide backdoor access to the compromised host at a later time, or can be used to direct the host to act as a zombie in attacks against other hosts. Although these new services can sometimes be detected by the output of system commands on the compromised host, those commands are also frequently modified. You may be able to detect new daemons using nmap or another port scanning tool from an uncompromised machine on the same network. (Of course, it’s always safest to disconnect a compromised machine from your network while you’re investigating it.)

You may also need to sweep the entire filesystem and observe what files and directories were accessed around the time of the intrusion. This may give you some clues as to what was done. For instance, if your compiler, loader, and libraries all show access times within a few seconds of each other, you can conclude that the intruder compiled something.

If you open files to search for changes, the time of last access on those files will change. Therefore, you will be unable to detect patterns of access. For this reason, we suggest that you conduct your forensics on a copy of your disks, mounted read-only. If you don’t have the hardware to make a copy, many systems will allow you to remount live partitions read-only (possibly through a loopback interface). Do your forensics on the copy. But realize that simply executing commands on this setup will likely change their access times, and the access times of any shared libraries and configuration files (unless you remount every partition)! Thus, your best bet may be to mount the disks read-only on another system, and do your checking from there.

Never Trust Anything Except Hardcopy

If your system is compromised, don’t trust anything that is on its disks. If you discover changes in files on your system that seem suspicious, don’t believe anything that your system tells you, because a good system cracker can change anything on the computer. The attacker can compile and install new versions of any system program—so there might be changes, but your standard utilities might not tell you about them. The attacker can patch the kernel that the computer is running, possibly disabling security features that you have previously enabled. The attacker can even open the raw disk devices for reading and writing. Often, they don’t need (or have) great skill. Instead, they have access to rootkits put together by others with more skill.

The only limit to the powers of an attacker who has gained superuser status is that the attacker cannot change something that has been indelibly printed on a piece of paper to which the attacker does not have access. For this reason, if you have a logging facility that logs whenever the date is changed, you might consider having the log made to a hardcopy terminal or to another computer. Then, be sure to examine this log on a regular basis.

To further protect yourself, you should have a bootable copy of your operating system on a zip disk, CD-ROM, or other removable storage device. This gives you a way of booting and examining your system with a set of tools that are known to be uncorrupted. Coupled with a database of message digests of unmodified files,245

you should be able

to find anything important that was modified on your system — provided that your message digests were generated from uncorrupted versions of your software. Remember that you cannot necessarily trust your backups, as you don’t know when the intrusion started: use distribution media if possible.

The next step is to get a printed copy of all of the necessary logs that you may have available (e.g., console logs and printed copies of network logs), and to examine these logs to try to get an idea of what the unauthorized intruder has done. You will also want to see if anything unusual has happened on the system since the time the intruder logged in. These logs may give you a hint as to what programs the intruder was running and what actions the intruder took. Be sure to initial and timestamp these printouts.

Keep in mind that the time you discover a break-in is not necessarily the same time as when it started! In one incident, there was evidence that the actual intrusion had occurred two years before! There were no backups or copies of software on the system that could be trusted. In fact, the intruders had been making wholesale changes to the system during all that time … installing patches and upgrades! They were doing a better job of administration than the person charged with managing the machine.

Resuming Operation

The next step in handling a break-in is to restore the system to a working state. How quickly you must be back in operation, and what you intend to do about the break-in over the long term, will determine when and how you perform this step.

At a minimum, you want to get whatever assurance you can that you have restored anything damaged on the system, and fixed whatever it was that allowed the intruder in. Then, if you have been keeping good backups, you can restore the system to a working state.

The difficulty with determining what failed and allowed an intruder in is complicated by the fact that there is usually little data in the logs to show what happened, and there are few things you can execute to reverse-engineer the break-in. Most break-ins seem to result from either bugs or, less commonly, compromised user passwords (suspect this especially if you find that the intruders have installed a sniffer on your system).

If the break-in was the result of a bug, you may have difficulty determining what it is, especially if it is a new one that has not been widely exploited. Here are some things to try:

1. If you have been recording your network traffic, examine your analysis system to see if any of the traffic is odd or unexplained.

2. Examine your log files looking for unusual entries, unusual patterns of activity, or evidence that programs have crashed.

3. If you know the specific IP address that the attacker used as the source of the attack, search through all of your log files for records of that IP address.

245 You can use Tripwire to produce such a database, or you can develop your own software.

If you suspect that it is a bug in some system software, you can try contacting your vendor to see if you can get some assistance there. In most cases it helps if you have a maintenance contract or are a major customer.

You might consult recent postings to the security groups on web sites or mailing lists. Often, current vulnerabilities are discussed at these locations in great detail. It is also the case that sometimes these sites contain information that is incorrect and even dangerous advice. Therefore, be very wary of what you read.

Finally, you may wish to contact a FIRST team appropriate for your site. Teams in FIRST often have some insight into current break-ins, largely because they see so many reports of them. Contacting a representative from one of the teams may result in some good advice for things to check before you put your system back into operation. However, many teams have rules of operation that prevent them from giving too much explicit information about active vulnerabilities until the appropriate vendors have announced a fix. Thus, you may not be able to get complete information from this source.

Damage Control

If you’ve already restored the system, what damage is there to control? Well, the aftermath, primarily. You need to follow through on any untoward consequences of the break-in. For instance, was proprietary information copied? If so, you need to notify your legal counsel and consider what to do.

You should determine which of the following concerns need to be addressed:

1. Do you need to file a formal report with law enforcement, a regulatory agency, an insurance company, or your vendor?

2. Do you need to institute disciplinary or dismissal actions against one or more employees? Do you need to update employee training to forestall any future incidents of this type?

3. Do you need to update your disaster recovery plan to account for changes or experiences in this instance?

4. Do you need to investigate and fix the software or configuration of any other systems under your control, or at any affiliated sites?

5. Do you need to have your public relations office issue a formal report (inside or outside) about this incident?

The answers to the above questions will vary from situation to situation and incident to incident.