11.4 Memory Debugging

Unlike desktop Linux systems, embedded Linux systems cannot afford to let applications eat up memory as they go or generate dumps because of illegal memory references. Among other things, there is no user to stop the offending applications and restart them. In developing applications for your embedded Linux system, you can employ special debugging libraries to ensure their correct behavior in terms of memory use. The following sections discuss two such libraries, Electric Fence and MEMWATCH.

Though both libraries are worth linking to your applications during development, production systems should not include either library. First, both libraries substitute the C library's memory allocation functions with their own versions of these functions, which are optimized for debugging, not performance. Secondly, both libraries are distributed under the terms of the GPL. Hence, though you can use MEMWATCH and Electric Fence internally to test your applications, you cannot distribute them as part of your applications outside your organization if your applications aren't also distributed under the terms of the GPL.

11.4.1 Electric Fence

Electric Fence is a library that replaces the C library's memory allocation functions, such as malloc( ) and free( ), with equivalent functions that implement limit testing. It is, therefore, very effective at detecting out-of-bounds memory references. In essence, linking with the Electric Fence library will cause your applications to fault and dump core upon any out-of-bounds reference. By running your application within gdb, you can identify the faulty instruction immediately.

Electric Fence was written and continues to be maintained by Bruce Perens. It is available from http://perens.com/FreeSoftware/. Download the package and extract it in your ${PRJROOT}/debug directory. For my control module, for example, I used Electric Fence 2.1.

Move to the package's directory for the rest of the installation:

$ cd ${PRJROOT}/debug/ElectricFence-2.1

Before you can compile Electric Fence for your target, you must edit the page.c source file and comment out the following code segment by adding #if 0 and #endif around it:

#if ( !defined(sgi) && !defined(_AIX) )
extern int           sys_nerr;
extern char *        sys_errlist[  ];
#endif

If you do not modify the code in this way, Electric Fence fails to compile. With the code changed, compile and install Electric Fence for your target:

$ make CC=powerpc-linux-gcc AR=powerpc-linux-ar
$ make LIB_INSTALL_DIR=${TARGET_PREFIX}/lib \
> MAN_INSTALL_DIR=${TARGET_PREFIX}/man install

The Electric Fence library, libefence.a, which contains the memory allocation replacement functions, has now been installed in ${TARGET_PREFIX}/lib. To link your applications with Electric Fence, you must add the -lefence option to your linker's command line. Here are the modifications I made to my command module's Makefile:

CFLAGS        = -g -Wall
...
LDFLAGS       = -lefence

The -g option is necessary if you want gdb to be able to print out the line causing the problem. The Electric Fence library adds about 30 KB to your binary when compiled in and stripped. Once built, copy the binary to your target for execution as you would usually.

By running the program on the target, you get something similar to:

# command-daemon
 
  Electric Fence 2.0.5 Copyright (C) 1987-1998 Bruce Perens.
Segmentation fault (core dumped)

Since you can't copy the core file back to the host for analysis, because it was generated on a system of a different architecture, start the gdb server on the target and connect to it from the host using the target gdb. As an example, here's how I start my command daemon on the target for Electric Fence debugging:

# gdbserver 192.168.172.50:2345 command-daemon

And on the host I do:

$ powerpc-linux-gcc command-daemon
(gdb) target remote 192.168.172.10:2345
Remote debugging using 192.168.172.10:2345
0x10000074 in _start (  )
(gdb) continue
Continuing.
 
Program received signal SIGSEGV, Segmentation fault.
0x10000384 in main (argc=2, argv=0x7ffff794) at daemon.c:126
126             input_buf[input_index] = value_read;

In this case, the illegal reference was caused by an out-of-bounds write to an array at line 126 of file daemon.c. For more information on the use of Electric Fence, look at the ample manpage included in the package.

11.4.2 MEMWATCH

MEMWATCH replaces the usual memory allocation functions, such as malloc( ) and free( ), with versions that keep track of allocations. It is very effective at detecting memory leaks such as when you forget to free a memory region or when you try to free a memory region more than once. This is especially important in embedded systems, since there is no one to monitor the device to check that the various applications aren't using up all the memory over time. MEMWATCH isn't as efficient as Electric Fence, however, to detect pointers that go astray. It was unable, for example, to detect the faulty array write presented in the previous section.

MEMWATCH is available from its project site at http://www.linkdata.se/sourcecode.html. Download the package and extract it in your ${PRJROOT}/debug directory. MEMWATCH consists of a header and a C file, which must be compiled with your application. To use MEMWATCH, start by copying both files to your application's source directory:

$ cd ${PRJROOT}/debug/memwatch-2.69
$ cp memwatch.c memwatch.h ${PRJROOT}/project/command-daemon

Modify the Makefile to add the new C file as part of the objects to compile and link. For my command daemon, for example, I used the following Makefile modifications:

CFLAGS        = -O2 -Wall -DMEMWATCH -DMW_STDIO
...
OBJS          = daemon.o memwatch.o

You must also add the MEMWATCH header to your source files:

#ifdef MEMWATCH
#include "memwatch.h"
#endif /* #ifdef MEMWATCH */

You can now cross-compile as you would usually. There are no special installation instructions for MEMWATCH. The memwatch.c and memwatch.h files add about 30 KB to your binary once built and stripped.

When the program runs, it generates a report on the behavior of the program, which it puts in the memwatch.log file in the directory where the binary runs. Here's an excerpt of the memwatch.log generated by running my command daemon:

=  ==  ==  ==  ==  ==  == MEMWATCH 2.69 Copyright (C) 1992-1999 Johan Lindh =  ==  ==...
...
unfreed: <3> daemon.c(220), 60 bytes at 0x10023fe4          {FE FE FE ...
...
Memory usage statistics (global):
 N)umber of allocations made: 12
 L)argest memory usage      : 1600
 T)otal of all alloc(  ) calls: 4570
 U)nfreed bytes totals      : 60

The unfreed: line tells you which line in your source code allocated memory that was never freed later. In this case, 60 bytes are allocated at line 220 of daemon.c and are never freed. The T)otal of all alloc( ) calls: line indicates the total quantity of memory allocated throughout your program's execution. In this case, the program allocated 4570 bytes in total.

Look at the FAQ, README, and USING files included in the package for more information on the use of MEMWATCH and the output it provides.