Recipe 9.14 Examining Local Network Activities

9.14.1 Problem

You want to examine network use occurring on your local machine.

9.14.2 Solution

To print a summary of network use:

$ netstat --inet                 Connected sockets
$ netstat --inet --listening     Server sockets
$ netstat --inet --all           Both
# netstat --inet ... -p          Identify processes

To print dynamically assigned ports for RPC services:

$ rpcinfo -p [host]

To list network connections for all processes:

# lsof -i[TCP|UDP][@host][:port]

To list all open files for specific processes:

# lsof -p pid
# lsof -c command
# lsof -u username

To list all open files (and network connections) for all processes:

# lsof

To trace network system calls, use strace . [Recipe 9.15]

9.14.3 Discussion

Suppose you see a process with an unfamiliar name running on your system. Should you be concerned? What is it doing? Could it be surreptitiously transmitting data to some other machine on a distant continent?

To answer these kinds of questions, you need tools for observing network use and for correlating activities with specific processes. Use these tools frequently so you will be familiar with normal network usage, and equipped to focus on suspicious behavior when you encounter it.

The netstat command prints a summary of the state of networking on your machine, and is a good way to start investigations. The —inet option prints active connections:

$ netstat --inet
Active Internet connections (w/o servers)
Proto   Recv-Q Send-Q Local Address           Foreign Address         State      
tcp     0      240    myhost.example.com:ssh  client.example.com:3672 ESTABLISHED
tcp     0      0      myhost.example.com:4099 server.example.com:ssh  TIME_WAIT

This example shows inbound and outbound ssh connections; the latter is shutting down (as indicated by TIME_WAIT). If you see an unusually large number of connections in the SYN_RECV state, your system is probably being probed by a port scanner like nmap. [Recipe 9.13]

Add the —listening option to instead see server sockets that are ready to accept new connections (or use —all to see both kinds of sockets):

$ netstat --inet --listening
Active Internet connections (only servers)
Proto Recv-Q   Send-Q   Local Address   Foreign Address     State
tcp        0        0   *:ssh           *:*                 LISTEN
tcp        0        0   *:http          *:*                 LISTEN
tcp        0        0   *:814           *:*                 LISTEN
udp        0        0   *:ntp           *:*
udp        0        0   *:811           *:*

This example shows the ssh daemon, a web server (http), a network time server (which uses udp), and two numerical mystery ports, which might be considered suspicious. On a typical system, you would expect to see many more server sockets, and you should try to understand the purpose of each. Consider disabling services that you don't need, as a security precaution.

Port numbers for RPC services are assigned dynamically by the portmapper. The rpcinfo command shows these assignments:

$ rpcinfo -p | egrep -w "port|81[14]"
    program vers proto   port
     100007    2   udp    811  ypbind
     100007    1   udp    811  ypbind
     100007    2   tcp    814  ypbind
     100007    1   tcp    814  ypbind

This relieves our concerns about the mystery ports found by netstat.

You can even query the portmapper on a different machine, by specifying the hostname on the command line. This is one reason why your firewall should block access to your portmapper, and why you should run it only if you need RPC services.

The netstat -p option adds a process ID and command name for each socket, and the -e option adds a username.

Only the superuser can examine detailed information for processes owned by others. If you need to observe a wide variety of processes, run these commands as root.

The lsof command lists open files for individual processes, including network connections. With no options, lsof reports on all open files for all processes, and you can hunt for information of interest using grep or your favorite text editor. This technique can be useful when you don't know precisely what you are looking for, because all of the information is available, which provides context. The voluminous output, however, can make specific information hard to notice.

lsof provides many options to select files or processes for more refined searches. By default, lsof prints information that matches any of the selections. Use the -a option to require matching all of them instead.

The -i option selects network connections: lsof -i is more detailed than but similar to netstat —inet —all -p. The -i option can be followed by an argument of the form [TCP|UDP][@host][:port] to select specific network connections—any or all of the components can be omitted. For example, to view all ssh connections (which use TCP), to or from any machine:

# lsof -iTCP:ssh
COMMAND PID  USER     FD TYPE DEVICE SIZE NODE NAME
sshd     678 root     3u IPv4   1279       TCP  *:ssh (LISTEN)                       
sshd    7122 root     4u IPv4 211494       TCP  myhost:ssh->client:367  (ESTABLISHED)
sshd    7125 katie    4u IPv4 211494       TCP  myhost:ssh->client:3672 (ESTABLISHED)
ssh     8145 marianne 3u IPv4 254706       TCP  myhost:3933->server:ssh (ESTABLISHED)

Note that a single network connection (or indeed, any open file) can be shared by several processes, as shown in this example. This detail is not revealed by netstat -p.

Both netstat and lsof convert IP addresses to hostnames, and port numbers to service names (e.g., ssh), if possible. You can inhibit these conversions and force printing of numeric values, e.g., if you are have many network connections and some nameservers are responding slowly. Use the netstat —numeric-hosts or —numeric-ports options, or the lsof -n, -P, or -l options (for host addresses, port numbers, and user IDs, respectively) to obtain numeric values, as needed.

To examine processes that use RPC services, the +M option is handy for displaying portmapper registrations:

# lsof +M -iTCP:814 -iUDP:811
COMMAND  PID  USER  FD  TYPE  DEVICE  SIZE  NODE  NAME
ypbind   633  root  6u  IPv4    1202         UDP  *:811[ypbind]
ypbind   633  root  7u  IPv4    1207         TCP  *:814[ypbind] (LISTEN)
ypbind   635  root  6u  IPv4    1202         UDP  *:811[ypbind]
ypbind   635  root  7u  IPv4    1207         TCP  *:814[ypbind] (LISTEN)
ypbind   636  root  6u  IPv4    1202         UDP  *:811[ypbind]
ypbind   636  root  7u  IPv4    1207         TCP  *:814[ypbind] (LISTEN)
ypbind   637  root  6u  IPv4    1202         UDP  *:811[ypbind]
ypbind   637  root  7u  IPv4    1207         TCP  *:814[ypbind] (LISTEN)

This corresponds to rpcinfo -p output from our earlier example. The RPC program names are enclosed in square brackets, after the port numbers.

You can also select processes by ID (-p), command name (-c), or username (-u):

# lsof -a -c myprog -u tony
COMMAND  PID  USER  FD TYPE  DEVICE    SIZE  NODE   NAME
myprog  8387  tony cwd  DIR   0,15     4096  42329  /var/tmp
myprog  8387  tony rtd  DIR    8,1     4096      2       /
myprog  8387  tony txt  REG    8,2    13798  31551  /usr/local/bin/myprog
myprog  8387  tony mem  REG    8,1    87341  21296  /lib/ld-2.2.93.so
myprog  8387  tony mem  REG    8,1    90444  21313  /lib/libnsl-2.2.93.so
myprog  8387  tony mem  REG    8,1    11314  21309  /lib/libdl-2.2.93.so
myprog  8387  tony mem  REG    8,1   170910  81925  /lib/i686/libm-2.2.93.so
myprog  8387  tony mem  REG    8,1    10421  21347  /lib/libutil-2.2.93.so
myprog  8387  tony mem  REG    8,1    42657  21329  /lib/libnss_files-2.2.93.so
myprog  8387  tony mem  REG    8,1    15807  21326  /lib/libnss_dns-2.2.93.so
myprog  8387  tony mem  REG    8,1    69434  21341  /lib/libresolv-2.2.93.so
myprog  8387  tony mem  REG    8,1  1395734  81923  /lib/i686/libc-2.2.93.so
myprog  8387  tony  0u  CHR  136,3               2  /dev/pts/3
myprog  8387  tony  1u  CHR  136,3               2  /dev/pts/3
myprog  8387  tony  2u  CHR  136,3               2  /dev/pts/3
myprog  8387  tony  3r  REG    8,5        0  98315  /var/tmp/foo
myprog  8387  tony  4w  REG    8,5        0  98319  /var/tmp/bar
myprog  8387  tony  5u  IPv4 274331             TCP  myhost:2944->www:http (ESTABLISHED)

Note that the arrow does not indicate the direction of data transfer for network connections: the order displayed is always local->remote.

The letters following the file descriptor (FD) numbers show that myprog has opened the file foo for reading (r), the file bar for writing (w), and the network connection bidirectionally (u).

The complete set of information printed by lsof can be useful when investigating suspicious processes. For example, we can see that myprog's current working directory (cwd) is /var/tmp, and the pathname for the program (txt) is /usr/local/bin/myprog. Be aware that rogue programs may try to disguise their identity: if you find sshd using the executable /tmp/sshd instead of /usr/sbin/sshd, that is cause for alarm. Similarly, it would be troubling to discover a program called "ls" with network connections to unfamiliar ports!^[5]

^[5] Even ls can legitimately use the network, however, if your system uses NIS for user or group ID lookups. You need to know what to expect in each case.

9.14.4 See Also

netstat(8), rpcinfo(8), lsof(8).