10.4 Managing System-Log Files

Configuring and fine-tuning your system-logging facilities is extremely important for system security and general diagnostics. But if your logs grow too large and fill up their filesystem, all that work may come to naught.

As with syslog itself, most Linux distributions come with a preconfigured log-rotation scheme. As with syslog, while this scheme tends to work adequately for many users, it's too important a mechanism to take for granted: it behooves you to understand, periodically evaluate, and, if necessary, customize your log-management setup.

10.4.1 Log Management in Red Hat 7 and Debian 2.2: /sbin/logrotate

Both Red Hat 7 and Debian 2.2 use a binary program called logrotate to handle system-log growth. In fact, they use very similar implementations of logrotate: global options and low-level (system) log files are addressed in /etc/logrotate.conf, and application-specific configuration scripts are kept in /etc/logrotate.d/.

When logrotate is run, all scripts in /etc/logrotate.d are included into logrotate.conf and parsed as one big script. This makes logrotate's configuration very modular: when you install an RPM or DEB package (of software that creates logs), your package manager automatically installs a script in /etc/logrotate.d, which will be removed later if you uninstall the package.

Actually, the include directive in logrotate.conf may be used to specify additional or different directories and files to include. In no event, however, should you remove the statement that includes /etc/logrotate.d if you use Red Hat or Debian, both of whose package managers depend on this directory for package-specific log-rotation scripts.

10.4.1.1 Syntax of logrotate.conf and its included scripts

There are really only two types of elements in logrotate.conf and its included scripts: directives (i.e., options) and log-file specifications. A directive is simply a parameter or a variable declaration; a log-file specification is a group of directives that apply to a specific log file or group of log files.

In Example 10-25, we see a simple /etc/logrotate.conf file.

Example 10-25. Simple logrotate.conf file

# Very simple logrotate.conf file
   
# Global options: rotate logs monthly, saving four old copies and sending
# error-messages to root. After "rotating out" a file, touch a new one
   
monthly
rotate 4
errors root
create
   
# Keep an eye on /var/log/messages
/var/log/messages {
   size 200k
   create
   postrotate
      /bin/kill -HUP `cat /var/run/syslog-ng.pid 2> /dev/null` 2> /dev/null || true
   endscript
}

In Example 10-25, the global options at the top may be thought of as the default log-file specification. Any directive for a specific log file takes precedence over the global options. Accordingly, we see in this example that although by default logs are rotated once a month and that four archives will be kept, the file /var/log/messages will be rotated not on the basis of time, but on size.

However, the other global directives will still apply: four old copies will be kept; immediately after a log is renamed (which is how they're "rotated"), a newly empty current log file will be created ("touched"); and error messages will be emailed to root.

logrotate supports a large number of different directives, but in practice, you'll probably spend more time tweaking the subscripts placed in logrotate.d than you will writing scripts from scratch. With that in mind, Table 10-12 lists some commonly encountered logrotate directives. A complete list is provided in the manpage logrotate(8).

Table 10-12. Common logrotate directives

Directive	Description
/path/to/logfile { directive1 directive2 etc. }	Log file specification header/footer (i.e., "apply these directives to the file /path/to/logfile"). Whitespace is ignored. Applicable global directives are also applied to the log file, but when a given directive is specified both globally and locally (within a log file specification), the local setting overrules the global one.
rotate number	Tells logrotate to retain number old versions of the specified log file. Setting this to zero amounts to telling logrotate to overwrite the old log file.
daily | weekly | monthly | size=number_bytes	The criterion for rotating the specified file: either because one day or week or month has passed since the last rotation, or because the file's size has reached or exceeded number_bytes since the last time logrotate was run. Note that if number_bytes is a number, bytes are assumed; if expressed as a number followed by a lowercase "k," Kilobytes are assumed; if expressed as a number followed by a capital "M," Megabytes are assumed.
mail [username|mail@address]	Email old files to the specified local user or email address rather than deleting them.
errors [username|email@address]	Email logrotate error messages to the specified local user or email address.
compress	Use gzip to compress old versions of log files.
copytruncate	Instead of renaming the current log file and creating a new (empty) one, move most of its data out into an archive file. Accommodates programs that can't interrupt logging (i.e., that need to keep the log file open for writing continuously).
create [octalmode owner group]	Recreate the (now empty) log file immediately after rotation. If specified, set any or all of these properties: octalmode (file-mode in octal notation — e.g., 0700), owner, and group properties.
ifempty | notifempty	By default, logrotate will rotate a file even if it's empty. notifempty cancels this behavior; ifempty restores it (e.g., overriding a global notifempty setting).
include file_or_directory	When parsing logrotate.conf, include the specified file or the files in the specified directory.
missingok | nomissingok	By default, logrotate will return a message if a log file doesn't exist. nomissingok cancels this behavior (i.e., tells logrotate to skip that log file quietly); missingok restores the default behavior (e.g., overriding a global nomissingok setting).
olddir dir | noolddir	Tells logrotate to keep old versions of a log file in dir, whereas noolddir tells logrotate to keep old versions in the same directory as the current version (noolddir is the default behavior).
postrotate line1 line2 etc. endscript	Execute specified lines after rotating the log file. Can't be declared globally. Typically used to send a SIGHUP to the application that uses the log file.
prerotate line1 line2 etc. endscript	Execute specified lines before rotating the log file. Can't be declared globally.

Just What Do We Mean By "Rotate?" All log-management mechanisms involve periodically moving/renaming a log file to an archive copy and creating a new (empty) log file. Rotation is necessary when multiple archive copies are maintained. In the most common log-rotation scheme, a set of static filenames is maintained. For example, messages, messages.1, messages.2, messages.3 is a typical three-archive filename set — messages being the "current" log file and messages.3 being the oldest archive. In this scheme, rotation is achieved by coping the second-to-oldest file over the oldest file (e.g., mv messages.2 messages.3). The third-oldest file's name is then changed to that of the second-oldest file's, and so forth, until the current file is renamed and a new (empty) "current" log file is created (e.g., mv messages messages.1; touch messages). This is how logrotate behaves when its rotate parameter is set to a nonzero value. In the second common mechanism, archive filenames are unique (e.g., messages, messages.20010807, messages.20010708, etc.). In this case, rotation is a simple matter of changing the current file's name and then creating a new (empty) "current" log file (e.g., mv messages messages.20010928; touch messages). The final step is to compare the age of the oldest log archive file to a "maximum age" setting and to delete it if it's reached that age. This second scheme is used by SuSE's aaa_base_rotate_logs script (covered later in this chapter).

10.4.1.2 Running logrotate

In both Red Hat 7 and Debian 2.2, logrotate is invoked by the script /etc/cron.daily/logrotate, which consists of a single command:

/usr/sbin/logrotate /etc/logrotate.conf

This doesn't necessarily mean that logs are rotated daily; it means that logrotate checks each log file daily against its configuration script and rotates or doesn't rotate the log file accordingly.

If you want logrotate to be run less frequently, you can move this script to /etc/cron.weekly or even /etc/cron.monthly (though the latter is emphatically not recommended unless logrotate is, for some strange reason, configured to rotate each and every file monthly).

10.4.2 Log Management in SuSE 7

Log rotation in SuSE, as with so much else, is configured at a gross level in /etc/rc.config (the configuration file for suseconfig, which is the primary backend engine of yast). This file contains a variable called MAX_DAYS_FOR_LOG_FILES, which you can use to set the maximum number of days system logs are kept (by default, 365). In addition, the log-rotation tools themselves come preconfigured and preactivated.

Chances are, however, that you'll need to tweak SuSE's log-management setup more granularly than MAX_DAYS_FOR_LOG_FILES, especially if you install Syslog-ng and disable syslog. As it happens, SuSE's log-rotation scheme is less powerful but also much simpler than Red Hat's and Debian's logrotate.

SuSE uses a script called /etc/cron.daily/aaa_base_rotate_logs for day-to-day log rotation. This script shouldn't be manually edited; its behavior is controlled by the file /etc/logfiles, which is simply a list of the files you wish to rotate along with the maximum sizes you want them to reach, the permissions and ownerships they should have, and the startup script (if any) that should be restarted after rotation is done.

Example 10-26 is an excerpt from the default /etc/logfiles from SuSE 7.1.

Example 10-26. Excerpts from /etc/logfiles

# /etc/logfiles - This file tells cron.daily, which log files have to be watched
#
# File                       max size   mode     ownership    service 
#                                                           (reload if changed)
/var/log/mgetty.*              +1024k   644      root.root
/var/log/messages              +4096k   640      root.root
/var/log/httpd/access_log      +4096k   644      root.root     apache
/var/squid/logs/access.log     +4096k   640      squid.root

In the first noncomment line, all log files whose name begins /var/log/mgetty will be rotated after exceeding 1,024 kilobytes, after which they'll be rotated to new files whose permissions are -rw-r--r-- and that are owned by user root and group root.

The third line states that the file /var/log/httpd/access_log should be rotated after exceeding 4,096 kilobytes, should be recreated with permissions -rw-r--r--, owned by user root and group root, and after rotation is done, the startup script /etc/init.d/apache should be restarted.

Since the maximum age of all log files is set globally in /etc/rc.config, take care not to set the maximum size of a frequently written-to file (such as /var/log/messages) too high. If this happens and if the maximum age is high enough, your logs may fill their volume.

Speaking of which, I highly recommend the use of a dedicated /var partition on any machine that acts as a server; a full /var partition is much less likely to cause disruptive system behavior (e.g., crashing) than a full root partition.