10.5 Using Swatch for Automated Log Monitoring

Okay, you've painstakingly configured, tested, and fine-tuned your system logger to sort system messages by type and importance and then log them both to their respective files and to a central log server. You've also configured a log-rotation scheme that keeps as much old log data around as you think you'll need.

But who's got the time to actually read all those log messages?

swatch (the "Simple WATCHer") does. swatch, a free log-monitoring utility written 100% in Perl, monitors logs as they're being written and takes action when it finds something you've told it to look out for. Swatch does for logs what tripwire does for system-file integrity.

10.5.1 Installing Swatch

There are two ways to install swatch. First, of course, is via whatever binary package of swatch your Linux distribution of choice provides. (I use the term loosely here; "executable package" is more precise.) The current version of Mandrake has an RPM package of swatch, but none of the other most popular distributions (i.e., Red Hat, SuSE, Slackware, or Debian) appear to.

This is just as well, though, since the second way to install swatch is quite interesting. swatch's source distribution, available from http://www.stanford.edu/~atkins/swatch, includes a sophisticated script called Makefile.PL that automatically checks for all necessary Perl modules (see Should We Let Perl Download and Install Its Own Modules? later in this chapter) and uses Perl 5's CPAN functionality to download and install any modules you need; it then generates a Makefile that can be used to build swatch.

After you've installed the required modules, either automatically from swatch's Makefile.PL script or manually (and then running perl Makefile.PL), Makefile.PL should return the contents of Example 10-27.

Example 10-27. Successful Makefile.PL run

 [root@barrelofun swatch-3.0.1]# perl Makefile.PL 
Checking for Time::HiRes 1.12 ... ok
Checking for Date::Calc ... ok
Checking for Date::Format ... ok
Checking for File::Tail ... ok
Checking if your kit is complete...
Looks good
Writing Makefile for swatch
[root@barrelofun swatch-3.0.1]#

Once Makefile.PL has successfully created a Makefile for swatch, you can execute the following commands to build and install it:

    make
    make test
    make install
    make realclean

The make test command is optional but useful: it ensures that swatch can properly use the Perl modules we just went to the trouble of installing.

Should We Let Perl Download and Install Its Own Modules? The Comprehensive Perl Archive Network (CPAN) is a network of Perl software archives from around the world. Perl Version 5.6.x includes modules (CPAN and CPAN::FirstTime, among others) that allow it to fetch, verify the checksums of, and even use gcc to compile Perl modules from CPAN sites on the Internet. In-depth descriptions of CPAN and Perl's CPAN functionality are beyond this chapter's scope, but I have one hint and one warning to offer. First, the hint. To install the module Example::Module (not a real Perl module), you enter the command: perl -MCPAN -e "install Example::Module" If it's the first time you've used the -MCPAN flag, the module CPAN::FirstTime will be triggered and you'll be asked to choose from various options as to how Perl should fetch and install modules from CPAN. These are well-phrased questions with reasonable defaults. But do pay attention to the output while this command executes: the module you're installing may depend on other modules and may require you to go back and execute, e.g.: perl -MCPAN -e "install Example::PreRequisite" before making a second attempt at installing the first module. Now for the warning: using CPAN is neither more nor less secure than downloading and installing other software from any other Internet source. On the one hand, before being installed, each downloaded module is automatically checked against a checksum that incorporates a cryptographically strong MD5 hash. On the other hand, this hash is intended to prevent corrupt downloads from going unnoticed, not to provide security per se. Furthermore, even assuming that a given package's checksum probably won't be replaced along with a tampered-with module (a big assumption), all this protects against is the unauthorized alteration of software after it's been uploaded to CPAN by its author. There's nothing to stop an evil registered CPAN developer (anybody may register as one) from uploading hostile code along with a valid checksum. But of course, there's nothing to stop that evil developer from posting bad stuff to SourceForge or FreshMeat, either. Thus, if you really want to be thorough, the most secure way to install a given Perl module is to: Identify/locate the module on http://search.cpan.org. Follow the link to CPAN's page for the module. Download the module not from CPAN, but from its developer's official web site (listed under "Author Information" in the web page referred to earlier in Step 2). If available, also download any checksum or hash provided by the developer for the tarball you just downloaded. Use gpg, md5, etc. to verify that the tarball matches the hash. Unzip and expand the tarball, e.g., tar -xzvf groovyperlmod.tar.gz. If you're a Righteously Paranoid Kung-Fu Master or aspire to becoming one, review the source code for sloppiness and shenanigans, report your findings to the developer or the world at large, and bask in the open source community's awe and gratitude. (I'm being flippant, but open source code is truly open only when people bother to examine it!) Follow the module's building and installing directions, usually contained in a file called INSTALL and generally amounting to something like: perl ./Makefile.PL make make test make install Note that if the modules you need are being brought to your attention by swatch's Makefile.PL script, then to use the paranoid installation method, you'll want to write down the needed module names and kill that script (via plain old CONTROL-c) before installing the modules and rerunning swatch's Makefile.PL. Before I forget, there's actually a third way to install missing Perl modules: from your Linux distribution's FTP site or CDROM. While none approach CPAN's selection, most Linux distributions have packaged versions of the most popular Perl modules. Following are the modules you need for swatch and the packages that contain them in Red Hat 7 and Debian 2.2: Perl ModuleRed Hat 7 RPMDebian "deb" package Date::Calcperl-Date-Calclibdate-calc-perl Time::HiResperl-Time-HiReslibtime-hires-perl Date::Formatperl-TimeDatelibtimedate-perl File::Tailperl-File-Taillibfile-tail-perl None of this may seem terribly specific to swatch, and indeed it isn't, but it is important — more and more useful utilities are being released either as Perl modules or as Perl scripts that depend on Perl modules, so the chances are that swatch will not be the last Makefile.PL-based utility you install. Understanding some ramifications of all this module madness is worth the liter of ink I just spent on it, trust me.

10.5.2 swatch Configuration in Brief

Since the whole point of swatch is to simplify our lives, configuring swatch itself is, well, simple. swatch is controlled by a single file, $HOME/.swatchrc by default. This file contains text patterns, in the form of regular expressions, that you want swatch to watch for. Each regular expression is followed by the action(s) you wish to swatch to take whenever it encounters that text.

For example, suppose you've got an Apache-based web server and you want to be alerted any time someone attempts a buffer-overflow attack by requesting an extremely long filename (URL). By trying this yourself against the web server while tailing its /var/apache/error.log, you know that Apache will log an entry that includes the string "File name too long." Suppose further that you want to be emailed every time this happens. Example 10-28 shows what you'd need to have in your .swatchrc file.

Example 10-28. Simple entry in .swatchrc

watchfor /File name too long/
        mail addresses=mick\@visi.com,subject=BufferOverflow_attempt

As you can see, the entry begins with a watchfor statement, followed by a regular expression. If you aren't yet proficient in the use of regular expressions, don't worry: this can be as simple as a snippet of the text you want swatch to look for, spelled out verbatim between two slashes.

Swatch will perform your choice of a number of actions when it matches your regular expression. In this example, we've told swatch to send email to mick\@visi.com, with a subject of BufferOverflow_attempt. Note the backslash before the @ sign — without it, Perl will interpret the @ sign as a special character. Note also that if you want spaces in your subject-line, each space needs to be escaped with a backslash — e.g., subject=Buffer\ Overflow\ attempt.

Actions besides sending email include the ones in Table 10-13.

Table 10-13. Some actions swatch can take

Action (keyword)	Description
echo=normal, underscore, blue, inverse, etc.	Print matched line to console, with or without special text mode (default mode is "normal").
bell N	Echo the line to console, with "beep" sounded N times (default = 1).
exec command	Execute the command or script command.
pipe command	Pipe the line to the command command.
throttle HH:MM:SS	Wait for HH:MM:SS (period of time) after a line triggers a match, before performing actions on another match of the same expression. Helps prevent Denial of Service attacks via swatch (e.g., deliberately triggering huge numbers of swatch events in a short period).

For more details on configuring these and the other actions that swatch supports, see the swatch(1) manpage.

If you use Syslog-ng, you may be able to use some combination of match( ) filters, program( ) destinations, and pipe( ) destinations to achieve most of what swatch does. However, swatch's throttle parameter is an important advantage: whereas Syslog-ng acts on every message that matches a given filter, throttle gives swatch the intelligence to ignore repeated occurrences of a given event, potentially preventing minor events from becoming major annoyances.

Let's take that example a step further. Suppose in addition to being emailed about buffer-overflow attempts, you want to know whenever someone hits a certain web page, but only if you're logged on to a console at the time. In the same .swatchrc file, you'd add something like Example 10-29.

Example 10-29. An event that beeps and prints to console

watchfor /wuzza.html/
    echo=red
    bell 2

You will only see these messages and hear these beeps if you are logged on to the console in the same shell session from which you launched swatch. If you log out to go get a sandwich, when you return and log back in, you will no longer see messages generated by the swatch processes launched in your old session, even though those processes will still be running.

When in doubt, add either a "mail" action or some other non console-specific action (e.g., an "exec" action that triggers a script that pages you, etc.), unless, that is, the pattern in question isn't critical.

Alert readers have no doubt noticed that the scenario in the previous example will work only for Apache installations in which both errors and access messages are logged to the same file. We haven't associated different expressions with different watched files, nor can we. But what if you want swatch to watch more than one log file?

This is no problem. Although each .swatchrc file may describe only one watched file, there's nothing to stop you from running multiple instances of swatch, each with its own .swatchrc file. In other words, .swatchrc is the default, but not the required name for swatch configurations.

To split our two examples into two files, you'd put the lines in Example 10-27 into a file called, for example, .swatchrc.hterror and the lines in Example 10-28 into a file called .swatchrc.htaccess.

10.5.3 Advanced swatch Configuration

So far we've only considered actions we want triggered every time a given pattern is matched. There are several ways we can control swatch's behavior with greater granularity, however.

The first and most obvious is that search patterns take the form of regular expressions. Regular expressions, which really constitute a text-formatting language of their own, are incredibly powerful and responsible for a good deal of the magic of Perl, sed, vi, and many other Unix utilities.

It behooves you to know at least a couple "regex" tricks. Trick number one is called alternation, and it adds a "logical or" to your regular expression in the form of a "|" sign. Consider this regular expression:

/reject|failed/

This expression will match any line containing either the word "reject" or the word "failed." Use alternation when you want swatch to take the same action for more than one pattern.

Trick number two is the Perl-specific regular-expression modifier "case-insensitive," also known as "slash-i" since it always follows a regular expression's trailing slash. The regular expression:

/reject/i

matches any line containing the word "reject" whether it's spelled "Reject," "REJECT," "rEjEcT," etc. Granted, this isn't nearly as useful as alternation, and in the interest of full disclosure, I'm compelled to mention that slash-i is one of the more CPU-intensive Perl modifiers. However, if despite your best efforts at log tailing, self attacking, etc., you aren't 100% sure how a worrisome attack might look in a log file, slash-i helps you make a reasonable guess.

Another way to control swatch more precisely is to specify what time of day a given action may be performed. You can do this by sticking a when= option after any action. For example, in Example 10-30, I have a .swatchrc entry for a medium-importance event, which I want to know about via console messages during weekdays, but which I'll need email messages to know about during the weekend.

Example 10-30. Actions with when option specified

/file system full/
    echo=red
    mail addresses=mick\@visi.com,subject=Volume_Full,when=7-1:1-24

The syntax of the when= option is when=range_of_days:range_of_hours. Thus, in Example 10-30, we see that any time the message "file system full" is logged, swatch will echo the log entry to the console in red ink. It will also send email, but only if it's Saturday ("7") or Sunday ("1").

10.5.4 Running swatch

Swatch expects .swatchrc to live in the home directory of the user who invokes swatch. Swatch also keeps its temporary files there by default. (Each time it's invoked, it creates and runs a script called a "watcher process," whose name ends with a dot followed by the PID of the swatch process that created it).

The -c path/to/configfile and --script-dir=/path/to/scripts flags let you specify alternate locations for swatch's configuration and script files, respectively. Never keep either in a world-writable directory, however. In fact, only these files' owners should be able to read them.

For example, to invoke swatch so that it reads my custom configuration file in /var/log and also uses that directory for its watcher process script, I'd use the command listed in Example 10-31.

Example 10-31. Specifying nondefault paths

mylinuxbox:~# swatch -c /var/log/.swatchrc.access --script-dir=/var/log &

I also need to tell swatch which file to tail, and for that I need the -t filename flag. If I wanted to use the previous command to have swatch monitor /var/log/apache/access_log, it would look like this:

mylinuxbox:~# swatch -c /var/log/.swatchrc.access --script-dir=/var/log 
\  -t /var/log/apache/access_log &

swatch generally doesn't clean up after itself very well; it tends to leave watcher-process scripts behind. Keep an eye out and periodically delete these in your home directory or in the script directories you tend to specify with — script-dir.

Again, if you want swatch to monitor multiple files, you'll need to run swatch multiple times, with at least a different tailing target (-t value) specified each time and probably a different configuration file for each as well.

10.5.5 Fine-Tuning swatch

Once swatch is configured and running, we must turn our attention to the Goldilocks Goal: we want swatch to be running neither "too hot" (alerting us about routine or trivial events) nor "too cold" (never alerting us about anything). But what constitutes "just right?" There are as many answers to this question as there are uses for Unix.

Anyhow, you don't need me to tell you what constitutes nuisance-level reporting: if it happens, you'll know it. You may even experience a scare or two in responding to events that set off alarms appropriately but turn out to be harmless nonetheless. Read the manual, tweak .swatchrc, and stay the course.

The other scenario, in which too little is watched for, is much harder to address, especially for the beginning system administrator. By definition, anomalous events don't happen very frequently, so how do you anticipate how they'll manifest themselves in the logs? My first bit of advice is to get in the habit of browsing your system logs often enough to get a feel for what the routine operation of your systems looks like.

Better still, "tail" the logs in real time. If you enter the command tail -f /var/log/messages, the last 50 lines of the system log will be printed, plus all subsequent lines, as they're generated, until you kill tail with a Control-c. This works for any file, even a log file that changes very rapidly.

Another good thing you can do is to "beat up on" (probe/attack) your system in one virtual console or xterm while tailing various log files in another. nmap and Nessus, which are covered in Chapter 3 (Hardening Linux), are perfect for this.

By now you may be saying, "Hey, I thought the whole reason I installed swatch was so I wouldn't have to watch log files manually!" Wrong. Swatch minimizes, but does not eliminate, the need for us to parse log files.

Were you able to quit using your arithmetic skills after you got your first pocket calculator? No. For that matter, can you use a calculator in the first place unless you already know how to add, multiply, etc.? Definitely not. The same goes for log file parsing: you can't tell swatch to look for things you can't identify yourself, no more than you can ask for directions to a town whose name you've forgotten.

10.5.6 Why You Shouldn't Configure swatch Once and Forget About It

In the same vein, I urge you to not be complacent about swatch silence. If swatch's actions don't fire very often, it could be that your system isn't getting probed or misused very much, but it's at least as likely that swatch isn't casting its net wide enough. Continue to periodically scan through your logs manually to see if you're missing anything, and continue to tweak .swatchrc.

Don't forget to periodically reconsider the auditing/logging configurations of the daemons that generate log messages in the first place. Swatch won't catch events that aren't logged at all. Refer to the syslogd(8) manpage for general instructions on managing your syslogd daemon, and the manpages of the various things that log to syslog for specific instructions on changing the way they log events.