Sending Traps (Essential SNMP)

10.3.1. Sending Traps with OpenView

OpenView has a command-line program for generating arbitrary traps called snmptrap. snmptrap supports the counter, counter32, counter64,[50] gauge, gauge32, integer, integer32, ipaddress, null, objectidentifier, octetstring, octetstringascii, octetstringhex, octetstringoctal, opaque, opaqueascii, opaquehex, opaqueoctal, timeticks, and unsigned32 datatypes. Its command-line structure looks like this:

[50]This type will work only on agents that support SNMPv2.

snmptrap -c community [-p port] node_addr enterprise_id agent-addr generic \
specific timestamp [OID type value] ...

Here's a typical snmptrap command. It sends one trap, with three ASCII-string variable bindings for values:

$ /opt/OV/bin/snmptrap -c public nms \
.1.3.6.1.4.1.2789.2500 "" 6 3003 "" \ 
.1.3.6.1.4.1.2789.2500.3003.1 octetstringascii "Oracle" \
.1.3.6.1.4.1.2789.2500.3003.2 octetstringascii "Backup Not Running" \
.1.3.6.1.4.1.2789.2500.3003.3 octetstringascii "Call the DBA Now for Help"

It's a complicated command, and it's hard to imagine that you would ever type it on the command line. Let's break it up into pieces. The first line specifies the community string (public) and the address to which the trap should be sent (nms, though in practice it would be better to use an IP address rather than a node name). The next line is in many respects the most complicated. It specifies the enterprise ID for the trap we're going to send (.1.3.5.1.6.1.2789.2500, which is a subtree of the enterprise-specific tree we've devoted to traps); the address of the agent sending the trap (in this case, the null string "", which defaults to the agent's address; if you're using a proxy server, it is useful to specify the agent's address explicitly); the generic trap number (6, which is used for all enterprise-specific traps); the specific trap number (3003, which we've assigned); and a timestamp ("", which defaults to the current time).

The remaining three lines specify three variable bindings to be included with the trap. For each binding, we have the variable's object ID, its datatype, and its value. The variables we're sending are defined in our private (enterprise-specific) MIB, so their OIDs all begin with .1.3.6.1.4.1.2789.2500. All the variables are strings, so their datatype is octetstringascii. The trap PDU will be packed with these three strings, among other things. The program that receives the trap will decode the trap PDU and realize that there are three variable bindings in the trap. These variable bindings, like the one that reads "Call the DBA Now for Help," can be used to alert the operator that something bad has happened.

10.3.2. Sending Traps with Perl

In Chapter 8, "Polling and Setting" we learned how to use the get and set pieces of the SNMP Perl module. In this section we'll see how to use the snmptrap( ) routine to generate traps. Currently, SNMP_util supports only three types for traps: string, int, and oid. This can seem limiting, but it covers most needs. Here's how snmptrap is called:

snmptrap(communityname@host:port_number, enterpriseOID, host_name_from, \
generic_ID, specific_ID, OID, type, value, [OID, type, value ...])

One call to snmptrap can include any number of values; for each value, you must specify the object ID, the datatype, and the value you're reporting. The next script generates a trap with only one value:

#!/usr/local/bin/perl
# Filename: /opt/local/perl_scripts/snmptrap.pl 

use SNMP_util "0.54";  # This will load the BER and SNMP_Session for us

snmptrap("public\@nms:162", ".1.3.6.1.4.1.2789", "sunserver1", 6, 1247, \
         ".1.3.6.1.4.1.2789.1247.1", "int", "2448816");

The call to snmptrap( ) sends a trap to port 162 on host nms. The trap is sent from host sunserver1; it contains a single variable binding, for the object .1.3.6.1.4.1.2789.1247.1. The OID's type is int and its value is 2448816.

Now let's try sending a trap with multiple values (multiple variable bindings). The first object we'll report is an integer, to which we give the arbitrary value 4278475. The second object has a string value and is a warning that our database has stopped. Because we're using OIDs that belong to our own enterprise, we can define these objects to be anything we want:

snmptrap("public\@nms:162", ".1.3.6.1.4.1.2789", "sunserver2", 6, 3301, \
         ".1.3.6.1.4.1.2789.3301.1", "int",    "4278475", \
         ".1.3.6.1.4.1.2789.3301.2", "string", "Sybase DB Stopped");

We can use the Net-SNMP snmptrapd program to monitor the traps coming in. We executed the preceding Perl code while running snmptrapd in stdout mode, and received:

$ ./snmptrapd -P
1999-10-12 09:45:08  [12.1.45.26] enterprises.2789.3000:
        Enterprise Specific Trap (3301) Uptime: 0:00:00
        enterprises.2789.3301.1 = 4278475
        enterprises.2789.3301.2 = "Sybase DB Stopped"

snmptrapd reported both of the values we sent in the trap: we see the integer value 4278475 and the notification that Sybase has stopped. Although this example is highly artificial, it's not all that different from what you would do when writing your own monitoring software. You would write whatever code is necessary to monitor vital systems such as your database and use the Perl SNMP module to send traps when significant events occur. You can then use any program capable of receiving traps to inform you when the traps arrive. If you want, you can add logic that analyzes the values sent in the trap or takes other actions, such as notifying an operator via a pager.

10.3.3. Sending Traps with Network Computing Technologies Trap Generator

This Windows-based command-line utility gives us the same features as its Unix counterparts. It understands the String, Counter, Gauge, Integer, Address, OID, and TimeTicks datatypes. The command line for nttrapgen looks like this:

nttrapgen.exe -d DestinationIpAddress:port -c CommunityName
              -o senderOID -i senderIP -g GenericTrapType 
              -s SpecificTrapType -t timestamp -v OID TYPE VALUE

Here's how to use nttrapgen to send a trap notifying us that the UPS battery is running low. We use the String datatype to send an informative message, and we use trap 4025.1 from our private enterprise ID, 2789:

C:\tools> nttrapgen.exe -d nms:162 -c public -o ^ 
1.3.6.1.4.1.2789.4025 -i 10.123.456.4 -g 6 -s 4025 -t 124501 ^
-v 1.3.6.1.4.1.2789.4025.1 STRING 5 Minutes Left On UPS Battery

This trap will be sent to our network-management station (which has the hostname nms) on port 162, which is the standard port for SNMP traps. Any management station should be able to receive the trap and act on it appropriately. You can use this command in batch scripts, which are essentially the same as Unix shell scripts. Therefore, you can use nttrapgen to generate traps as you need them: you can write scripts that monitor key processes and generate traps when any interesting events take place. As with the earlier Perl example, you can use this simple trap generator in your environment if you don't need a heavy-duty management system.

10.3.4. Sending Traps with Net-SNMP

This snmptrap program looks very similar to OpenView's snmptrap. This program uses a single letter to refer to datatypes, as shown in Table 10-2.

Table 10-2. Net-SNMP snmptrap Datatypes

Abbreviation	Datatype
a	IP address
c	Counter
d	Decimal string
i	Integer
n	Null
o	Object ID
s	String
t	Time ticks
u	Unsigned integer
x	Hexadecimal string

Here's how the Net-SNMP snmptrap program is invoked:

snmptrap hostname community enterprise-oid agent \
generic-trap specific-trap uptime [OID type value]...

If you use two single quotes ('') in place of the time, snmptrap inserts the current time into the trap. The following command generates a trap with a single value. The object ID is 2005.1, within our private enterprise; the value is a string that tells us that the web server has been restarted:

$ snmptrap nms public .1.3.6.1.4.1.2789.2005 ntserver1 6 2476317 '' \
.1.3.6.1.4.1.2789.2005.1 s "WWW Server Has Been Restarted"

Here's how to send a Version 2 notification with Net-SNMP:[51]

[51]For information about sending Version 3 notifications with Net-SNMP, see Appendix F, "SNMPv3".

$ snmptrap -v2c nms public '' .1.3.6.1.6.3.1.1.5.3 \
ifIndex i 2 ifAdminStatus i 1 ifOperStatus i 1

The command is actually simpler than its Version 1 equivalent. There are no generic numbers, specific numbers, or vendor IDs. The "" argument defaults to the current system uptime. The OID specifies the linkDown notification, with three data bindings specifying the link's status. The definition of linkDown in the IF-MIB states that the linkDown notification must include the ifIndex, ifAdminStatus, and ifOperStatus objects, which report the index of the interface that went down, its administrative status, and its operational status, respectively. For ifAdminStatus and ifOperStatus, a value of 1 indicates that the link is up. So this notification reports that interface 2 has changed its state from "down" to "up."

Again, the snmptrap command-line tool lets you integrate SNMP monitoring into shell scripts and other programs.

10.3.5. Forcing Your Hardware to Generate Traps

When you install a new piece of equipment, you should verify that it generates traps correctly. Testing your equipment's ability to generate traps has the added benefit of testing the behavior of your NMS; you can ensure that it handles traps in the way you want. The best way to test new hardware is to read your vendor's MIB and look for all the TRAP-TYPEs they have defined. This will give you a good feel for what sort of traps your vendor has implemented. For example, I read through our APC MIB and noticed that the unit will send a trap when it goes onto battery power if the AC power goes out. To test this feature, I secured the area in our datacenter and switched off the circuit breaker to simulate a power failure. The trap was generated, but it showed up in the Error event category because I did not have the correct MIB loaded in OpenView. I took the OID from the Error events and searched the APC MIBs for a match. When I found one, I loaded the MIB file into OpenView and repeated the test. This time, when the trap was received OpenView put an informative message in the Event Categories.

Most SNMP-compatible routers, switches, and network devices can generate linkDown traps. From RFC 1157, a linkDown trap is a "failure in one of the communication links represented in the agent's configuration." This means that if you start unplugging ports on your router you should receive traps, right? Yes, but first make sure you don't start disconnecting production database servers. Furthermore, make sure you don't disconnect the port by which your device would send the trap back to the NMS. Remember, SNMP is designed with the assumption that the network is unreliable -- if something sends a trap but there's no way for the trap to reach its destination, no one will find out. By default, a linkDown trap won't appear in OpenView's Event Categories, because the default setting for linkDown is "Log only"; watch the log file $OV_LOG/trapd.log to see these traps arrive. Once you have a mechanism for receiving traps, bringing the link up and down on your device should send some traps your way.

10.3.6. Using Hooks with Your Programs

A hook is a convenient interface that lets you integrate your own code into some other product. The Emacs text editor is a good example of a program that uses hooks, almost entirely, to allow its users to extend how it operates. Let's look at the following simple program to explain this concept further:

# Logical Sample Program NH1
# PROGRAM COMMENTS
# PROGRAM BEGINS

		PROGRAM ADDS           $VAR1 + $VAR2 = $VAR3
		PROGRAM SUBTRACTS      $VAR5 - $VAR6 = $VAR7
		PROGRAM PRINTS RESULTS $VAR3 $VAR7

# PROGRAM ENDS

This program simply ADDS, SUBTRACTS, and PRINTS RESULTS; it does not have any hooks. To add a feature, you have to modify the code. For a small program like this that is a trivial exercise, but it would be difficult in a program of any size. The next program contains some hooks that let you add extensions:

# Logical Sample Program H1
# PROGRAM COMMENTS
# PROGRAM BEGINS
   PROGRAM RUNS $PATH/start.sh

   PROGRAM ADDS           $VAR1 + $VAR2 = $VAR3
   PROGRAM SUBTRACTS      $VAR5 - $VAR6 = $VAR7
   PROGRAM PRINTS RESULTS $VAR3 $VAR7

   PROGRAM RUNS $PATH/end.sh
# PROGRAM ENDS

Notice the two additional RUNS statements. These hooks allow you to run anything you want at the start or end of the program. The first program, start.sh, might be as simple as the command echo "I am starting", which sends a simple message to the system or management console. This script could also call one of the trap-generation programs to send a trap to the NMS stating that some program is starting. It would be even more useful to send a message when the program terminates, possibly including information about the program's status. Here's a slightly more complicated program that runs a script, providing a number of arguments so that the script can send useful information back to the NMS when it generates a trap:

# Logical Sample Program H2
# PROGRAM COMMENTS
# PROGRAM BEGINS
   PROGRAM RUNS $PATH/start.sh $PROGRAM_NAME

   PROGRAM ADDS           $VAR1 + $VAR2 = $VAR3
   PROGRAM SUBTRACTS      $VAR5 - $VAR6 = $VAR7
   PROGRAM PRINTS RESULTS $VAR3 $VAR7

   PROGRAM RUNS $PATH/end.sh $PROGRAM_NAME $VAR1 $VAR2 $VAR3 $VAR5 $VAR6 $VAR7
# PROGRAM ENDS

With the additional arguments available to the hook programs, we can generate messages like "The Program Widget has ended with sales at $4 and YTD at $7." If your hook programs are shell scripts, you can simply add snmptrap commands via a text editor. Once you finish adding the snmptrap code, you can test your hook program by running it on the command line.

On most systems, many scripts can benefit from snmptrap hooks. On Solaris or Linux machines, for example, some of your /etc/init.d scripts can be retrofitted to make use of snmptrap commands. It might be useful to have some kind of notification when important processes such as your web server or DNS server start and stop. Having such information on hand might make life much easier for your helpdesk. (The Concord SystemEDGE SNMP agent provides more rigorous process-monitoring capabilities. See Chapter 11, "Extensible SNMP Agents" for more information on this product.)

It's harder to add hooks to programs written in languages like C, because you need access to the source code as well as the ability to figure out where to place the hooks. Once you have identified where your hooks go and added them, you must recompile the source code. Some programs have hooks built in, allowing you to run external programs or RPCs. Check your program's documentation for the locations of these hooks. This is much more convenient than trying to build your own hooks into another program. Once you have established what these external programs are called, you can start writing your own traps or adding to existing ones.


10.2. Receiving Traps		11. Extensible SNMP Agents