Recipe 9.18 Searching for Strings in Network Traffic

9.18.1 Problem

You want to watch network traffic, searching for strings in the transmitted data.

9.18.2 Solution

Use ngrep.

To search for packets containing data that matches a regular expression and protocols that match a filter expression:

# ngrep [grep-options] regular-expression [filter-expression]

To search instead for a sequence of binary data:

# ngrep -X hexadecimal-digits [filter-expression]

To sniff packets and save them in a file:

# ngrep -O filename [-n count] [-d interface] [-s snap-length] \
  regular-expression [filter-expression]

To read and display the saved network trace data:

$ ngrep -I filename regular-expression [filter-expression]

9.18.3 Discussion

ngrep is supplied with SuSE but not Red Hat; however, it is easy to obtain and install if you need it. Download it from http://ngrep.sourceforge.net and unpack it:

$ tar xvpzf ngrep-*.tar.gz

compile it:

$ cd ngrep
$ ./configure --prefix=/usr/local
$ make

and install it into /usr/local as root:^[8]

^[8] We explicitly install in /usr/local, because otherwise the configure script would install into /usr, based on the location of gcc. We recommend /usr/local to avoid clashes with vendor-supplied software in /usr; this recommendation is codified in the Filesystem Hierarchy Standard (FHS), http://www.pathname.com/fhs. The configure script used for ngrep is unusual—such scripts typically install into /usr/local by default, and therefore do not need an explicit —prefix option. We also create the installation directories if they don't already exist, to overcome deficiencies in the make install command.

# mkdir -p /usr/local/bin /usr/local/man/man8
# make install

Sometimes we are interested in observing the data delivered by network packets, known as the payload. Tools like tcpdump [Recipe 9.16] and especially Ethereal [Recipe 9.17] can display the payload, but they are primarily designed for protocol analysis, so their ability to select packets based on arbitrary data is limited.^[9]

^[9] The concept of a packet's payload is subjective. Each lower-level protocol regards the higher-level protocols as its payload. The highest-level protocol delivers the user data; for example, the files transferred by FTP.

The ngrep command searches network traffic for data that matches extended regular expressions, in the same way that the egrep command (or grep -E) searches files. In fact, ngrep supports many of the same command-line options as egrep, such as -i (case-insensitive), -w (whole words), or -v (nonmatching). In addition, ngrep can select packets using the same filter expressions as tcpdump. To use ngrep as an ordinary packet sniffer, use the regular expression ".", which matches any nonempty payload.

ngrep is handy for detecting the use of insecure protocols. For example, we can observe FTP transfers to or from a server, searching for FTP request command strings to reveal usernames, passwords, and filenames that are transmitted as clear text:

$ ngrep -t -x 'USER|PASS|RETR|STOR' tcp port ftp and host server.example.com
interface: eth0 (10.44.44.0/255.255.255.0)
filter: ip and ( tcp port ftp )
match: USER|PASS|RETR|STOR
#############
T 2003/02/27 23:31:20.303636 10.33.33.33:1057 -> 10.88.88.88:21 [AP]
  55 53 45 52 20 6b 61 74    69 65 0d 0a                USER katie..
#####
T 2003/02/27 23:31:25.315858 10.33.33.33:1057 -> 10.88.88.88:21 [AP]
  50 41 53 53 20 44 75 6d    62 6f 21 0d 0a             PASS Dumbo!..
#############
T 2003/02/27 23:32:15.637343 10.33.33.33:1057 -> 10.88.88.88:21 [AP]
  52 45 54 52 20 70 6f 6f    68 62 65 61 72 0d 0a       RETR poohbear..
########
T 2003/02/27 23:32:19.742193 10.33.33.33:1057 -> 10.88.88.88:21 [AP]
  53 54 4f 52 20 68 6f 6e    65 79 70 6f 74 0d 0a       STOR honeypot..
###############exit
58 received, 0 dropped

The -t option adds timestamps; use -T instead for relative times between packets. The -x option prints hexadecimal values in addition to the ASCII strings.

ngrep prints a hash character (#) for each packet that matches the filter expression: only those packets that match the regular expression are printed in detail. Use the -q option to suppress the hashes.

To search for binary data, use the -X option with a sequence of hexadecimal digits (of any length) instead of a regular expression. This can detect some kinds of buffer overflow attacks, characterized by known signatures of fixed binary data.

ngrep matches data only within individual packets. If strings are split between packets due to fragmentation, they will not be found. Try to match shorter strings to reduce (but not entirely eliminate) the probability of these misses. Shorter strings can also lead to false matches, however—a bit of experimentation is sometimes required. dsniff does not have this limitation. [Recipe 9.19]

Like other packet sniffers, ngrep can write and read libpcap-format network trace files, using the -O and -I options. [Recipe 9.16] This is especially convenient when running ngrep repeatedly to refine your search, using data captured previously, perhaps by another program. Usually ngrep captures packets until killed, or it will exit after recording a maximum number of packets requested by the -n option. The -d option selects a specific interface, if your machine has several. By default, ngrep captures entire packets (in contrast to tcpdump and ethereal), since ngrep is interested in the payloads. If your data of interest is at the beginning of the packets, use the -s option to reduce the snapshot and gain efficiency.

When ngrep finds an interesting packet, the adjacent packets might be of interest too, as context. The ngrep -A option prints a specified number of extra (not necessarily matching) packets for trailing context. This is similar in spirit to the grep -A option, but ngrep does not support a corresponding -B option for leading context.

A recommended practice: Save a generous amount of network trace data with tcpdump, then run ngrep to locate interesting data. Finally, browse the complete trace using Ethereal, relying on the timestamps to identify the packets matched by ngrep.

9.18.4 See Also

ngrep(8), egrep(1), grep(1), tcpdump(8). The home page for ngrep is http://ngrep.sourceforge.net, and the tcpdump home page is http://www.tcpdump.org.

Learn more about extended regular expressions in the O'Reilly book Mastering Regular Expressions.