Chapter 6. Automation and Testing

My test programs are intended to break the system, to push it to its extreme limits, to pile complication on complication, in ways that the system programmer never consciously anticipated. To prepare such test data, I get into the meanest, nastiest frame of mind that I can manage, and I write the cruelest code I can think of; then I turn around and embed that in even nastier constructions that are almost obscene.

—Donald Knuth, The Errors of T_EX, 1989

We frequently conduct seminars and briefings for software developers and security engineers from such industries as finance, communications, and pharmaceuticals. During each seminar, we ask the audience what software development processes and methodologies are in use at their particular companies. What do you think they tell us?

Without exception, the developers describe to us processes that, at best, can be called ad hoc. This discovery frightens us. If you want to build secure, reliable, and robust software, you simply can't do it without sticking to some kind of structured development methodology.

In previous chapters we examined the many ways that a program can come to contain security flaws throughout the various phases of its development and deployment lifecycle. In this chapter, we address the final step in putting that knowledge to practical use: testing and other means of automating parts of the development process. While you need to know what mistakes to avoid and what safe practices to follow, it's equally important that you know how to apply all of that knowledge to ensure that your software is actually more secure.

Let's return to the SYN flood example we've been looking at throughout this book. How might things have been different if the original designers of the TCP stack had done more security testing of their design and implementations? For that matter, what kind of testing could they have done? Because the principal flaw that led to the attacks was in the design, what could the designers have done to (more) exhaustively test their design and possibly find this flaw at a much earlier stage? We'll address the answers to these questions as we discuss testing and automation processes.

In this chapter, we don't try to tell you everything you need to know about the discipline of general application testing. For one thing, it would take too long. For another, we don't claim expertise in that broad discipline; what we do understand well enough to expound on is security testing. So in this chapter we look at methods and types of tools that you can apply to your job to help you consistently produce high-quality, secure software.

As in previous chapters, we walk through our topic in the same order a developer might proceed: design (from an architecturally secure basis), followed by implementation, followed by operations. Along the way, we'll emphasize the use of automated tools. As we mentioned in the Preface, we don't spend much time reviewing specific tools here, because such evaluations age quickly. As elsewhere in the book, we focus on the overall concepts and approaches that you will always find useful. For details about specific tools, we invite you to visit our web site at www.securecoding.org.

The practical case studies at the end of this chapter help illustrate how to best make use of the tools and methodologies presented here. These examples, which in some cases span the entire development lifecycle described in the book, are drawn from our many experiences reviewing the security of hundreds of applications.