7.3 Objective 3: Configure and Use System Log Files

Many events occur on your Linux system that should be logged for administrative purposes. Linux uses the syslog system to display and record messages describing these events. This system allows finely controlled logging of messages from the kernel as well as processes running on your system and remote systems. Messages can be placed on the console display, in log files, and on the text screens of users logged in to the system.

7.3.1 Configuring syslog

The behavior of syslog is controlled by its configuration file, /etc/syslog.conf. This text file contains lines indicating what is to be logged and where. Each line contains directives in this form:

facility.level action

The directives are defined as follows:

facility

This represents the creator of the message (that is, the kernel or a process) and is one of the following: auth, authpriv, cron, daemon, kern, lpr, mail, mark, news, syslog, user,orlocal0throughlocal7. The use of these facility designators allows you to control the destination of messages based on their origin. Facilities local0 through local7 are for any use you may wish to assign to them in your own programs and scripts.^[6]

^[6] It's possible that your distribution has assigned one or more of the local facilities already. Check your configuration before using a local facility.

level

Specifies a severity threshold beyond which messages are logged, and is one of the following (from lowest to highest severity): debug, info, notice, warning, err, crit, alert, oremerg. There is also a special level called none that will disable a facility. The level defines the amount of detail recorded in the log file. A single period separates the facility from the level, and together they comprise the message selector. The asterisk (*) can be used to describe all facilities or all levels.

action

The action directive is arguably misnamed. It represents the destination for messages that correspond to a given selector ( facility.level ). The action can be a filename (including the full pathname), a hostname preceded by the @ sign, or a comma-separated list of users or asterisk (this means all logged-in users will be included). The action is to send the message to the specified destination.

For example, if you wanted to create a separate log file for activity reported by the scripts you write, you might include a line like this in /etc/syslog.conf :

# Define a new log file for the local5 facility
local5.*                           /var/log/local5

You could then use the logger utility to write messages to the facility from your shell script:^[7]

^[7] syslog must be restarted or signaled to reinitialize before the new log file is created.

$ logger -p local5.info "Script terminated normally"

The message "Script terminated normally" would be placed into /var/log/local5, along with a timestamp and the hostname that sent the message. Example 7-3 contains an example /etc/syslog.conf file.

Example 7-3. Sample /etc/syslog.conf File

# Log everything except mail & authpriv of level info 
# or higher to messages
*.info;mail.none;authpriv.none     /var/log/messages

# The authpriv file has restricted access.
authpriv.*                         /var/log/secure

# Log all the mail messages in one place.
mail.*                             /var/log/maillog

# Everybody gets emergency messages
*.emerg                            *

# Save boot messages also to boot.log
local7.*                           /var/log/boot.log

On the Exam If you're not yet familiar with syslog, spend some time with it, modifying /etc/syslog.conf and directing messages to various files. An understanding of syslog is critical because so many programs depend on it.

If you examine this syslog.conf file, you'll see that nearly all system messages are sent to the /var/log/messages file via the *.info message selector. In this case, the asterisk directs syslog to send messages from all facilities except mail and authpriv, which are excluded using the special none level. The /var/log/messages file is the default system message destination, and you will consult it frequently for information on processes running (or failing to run) and other events on your system. In this example, the low severity level of info is used for the messages file, which logs all but debugging messages. On heavily loaded servers, this may result in an unwieldy file size due to message volume. Depending upon your available disk space, you may choose to save less information by raising the level for the messages file.

7.3.2 Log File Rotation

Most distributions will install a default syslog configuration for you, including logging to messages and other log files in /var/log. To prevent any of these files from growing unattended to extreme sizes, a log file rotation scheme should be installed as well. The cron system issues commands on a regular basis (usually once per day) to establish new log files; the old files are renamed with numeric suffixes (see Objective 4 for more on cron). With this kind of rotation, yesterday's /var/log/messages file becomes today's messages.1, and a new messages file is created. The rotation is configured with a maximum number of files to keep, and the oldest log files are deleted when the rotation is run.

The utility that establishes the rotation is logrotate. This privileged command is configured using one or more files, which are specified as arguments to the logrotate command. These configuration files can contain directives to include other files as well. The default configuration file is /etc/logrotate.conf. Example 7-4 depicts an example logrotate.conf file.

Example 7-4. Sample /etc/logrotate.conf File

# global options
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# send errors to root
errors root

# create new (empty) log files after rotating old ones
create

# compress log files 
compress

# specific files

/var/log/wtmp {
    monthly
    create 0664 root utmp
    rotate 1
}

/var/log/messages {
    postrotate
        /usr/bin/killall -HUP syslogd
    endscript
}

This example specifies rotations for two files, /var/log/wtmp and /var/log/messages. Your configuration will be much more complete, automatically rotating all log files on your system. A complete understanding of logrotate configuration is not necessary for LPIC Level 1 exams, but you must be familiar with the concepts involved. See the logrotate manpages for more information.

7.3.3 Examining Log Files

You can learn a lot about the activity of your system by reviewing the log files it creates. At times, it will be necessary to debug problems using logged information. Since most of the log files are plain text, it is very easy to review their contents with tools such as tail, less, and grep.

syslog stores the messages it creates with the following information, separated by (but also including) spaces:

• Date/time

• Origin hostname

• Message sender (such as kernel, sendmail, or a username)

• Message text

Typical messages will look like this:

Dec  8 10:41:23 smp kernel: Symbols match kernel 
      version 2.2.5.
Dec  8 10:41:23 smp kernel: Loaded 182 symbols 
      from 12 modules.
Dec  8 10:50:19 smp kernel: Kernel logging (proc) stopped.
Dec  8 10:50:19 smp kernel: Kernel log daemon terminating.

In this case, smp is the hostname, and the messages are coming from the kernel. At any time you can review the entire contents of your log files using less:

# less /var/log/messages

You can then page through the file. This is a good way to become familiar with the types of messages you'll see on your system. To actively monitor the output to your messages file, you could use tail:

# tail -f /var/log/messages

This might be useful, for example, to watch system activity as an Internet connection is established via modem. To look specifically for messages regarding your mouse, you might use grep:

# grep '[Mm]ouse' /var/log/messages
Dec  8 00:15:28 smp kernel: Detected PS/2 Mouse Port. 
Dec  8 10:55:02 smp gpm: Shutting down gpm mouse services: 

Often, if you are using grep to look for a particular item you expect to find in /var/log/messages, you will need to search all of the rotated files with a wildcard. For example, to look for all messages from sendmail, you may issue a command like this:

# grep 'sendmail:' /var/log/messages*

When you note problems in log files, look at the hostname and sender of the message first, then the message text. In many cases, you will be able to determine what is wrong from the message. Sometimes the messages are only clues, so a broader review of your logs may be necessary. In this case, it may be helpful to temporarily turn on more messaging by using the debug level in /etc/syslog.conf to help yield additional information that can lead you to the problem.