12.12 Detecting Debuggers

12.12.1 Problem

Software protection crackers frequently rely on debuggers to observe the runtime behavior of an application and to test binary patches that remove or bypass a protection. You would like to prevent casual analysis of your application by including anti-debugger code.

12.12.2 Solution

The Intel x86 instruction set uses the int3 opcode (0xCC) as a one-byte embedded breakpoint. Key addresses in the program—such as the first address in a function—can be checked to see whether they have been replaced with an int3 opcode.

12.12.3 Discussion

General debugger detection is difficult to perform successfully because of the limited number of techniques available and the ease with which they may be defeated. We advise you to attempt to detect specific debuggers in addition to using these general methods (see Recipe 12.13, Recipe 12.14, and Recipe 12.15).

The two macros defined below can be used to mark locations in the source where you might expect an int3 to be placed by someone trying to debug your program. The names used with these macros can then be passed as an argument to spc_check_int3( ) to test for the existence of the breakpoint instruction.

#define SPC_DEFINE_DBG_SYM(name) asm(#name ": \n")
#define SPC_USE_DBG_SYM(name)    extern void name(void)
   
inline int spc_check_int3(void *address) {
  return (*(volatile unsigned char *)address =  = 0xCC);
}

The SPC_DEFINE_DBG_SYM macro can be used to label an arbitrary code address, which can then be made available with the SPC_USE_DBG_SYM macro and passed to spc_check_int3( ):

#include <stdio.h>
   
void my_func(void) {
  int x;
   
  SPC_DEFINE_DBG_SYM(myfunc_nodebug);
  for (x = 0;  x < 10;  x++) printf("X!\n");
}
   
SPC_USE_DBG_SYM(myfunc_nodebug);
   
int main(int argc, char *argv[  ]) {
  if (spc_check_int3(myfunc_nodebug)) printf("being debugged: int3!\n");
  return(0);
}

Checking for int3 opcodes is a crude and largely unreliable method. The comparison with the 0xCC byte is immediately obvious when examining the disassembly of the above source code:

8048328 <dbg_check_int3>:
8048328:       push   %ebp
8048329:       mov    %esp, %ebp
804832b:       sub    $4, %esp
804832e:       mov    8(%ebp), %eax
8048331:       mov    (%eax), %al
8048333:       cmp    $0xCC, %al
8048335:       jne    8048340 
8048337:       movl   $1, -4(%ebp)
804833e:       jmp    8048347 
8048340:       movl   $0, -4(%ebp)
8048347:       mov    -4(%ebp), %eax
804834a:       leave  
804834b:       ret

The compare instruction at address 8048333 is obviously checking for an embedded int3 instruction. A software protection cracker can neutralize this check either by changing the 0xCC byte in the compare instruction to another value (such as 0x90, the nop instruction) or by changing the conditional jump instruction at address 8048335 (opcode 0x75) to an unconditional jump instruction (opcode 0xEB). In addition, most modern debuggers support the use of the debug registers present in Intel x86 CPUs because the Pentium breakpoints set using these registers do not require the int3 instruction and will not be detected with this method.

12.12.4 See Also

Recipe 12.13, Recipe 12.14, Recipe 12.15