1.9 Performance Checklist

• Specify the required performance.

  • Ensure performance objectives are clear.

  • Specify target response times for as much of the system as possible.

  • Specify all variations in benchmarks, including expected response ranges (e.g., 80% of responses for X must fall within 3 seconds).

  • Include benchmarks for the full range of scaling expected (e.g., low to high numbers of users, data, files, file sizes, objects, etc.).

  • Specify and use a benchmark suite based on real user behavior. This is particularly important for multiuser benchmarks.

  • Agree on all target times with users, customers, managers, etc., before tuning.

• Make your benchmarks long enough: over five seconds is a good target.

  • Use elapsed time (wall-clock time) for the primary time measurements.

  • Ensure the benchmark harness does not interfere with the performance of the application.

  • Run benchmarks before starting tuning, and again after each tuning exercise.

  • Take care that you are not measuring artificial situations, such as full caches containing exactly the data needed for the test.

• Break down distributed application measurements into components, transfer layers, and network transfer times.

• Tune systematically: understand what affects the performance; define targets; tune; monitor and redefine targets when necessary.

  • Approach tuning scientifically: measure performance; identify bottlenecks; hypothesize on causes; test hypothesis; make changes; measure improved performance.

  • Determine which resources are limiting performance: CPU, memory, or I/O.

  • Accurately identify the causes of the performance problems before trying to tune them.

  • Use the strategy of identifying the main bottlenecks, fixing the easiest, then repeating.

  • Don't tune what does not need tuning. Avoid "fixing" nonbottlenecked parts of the application.

  • Measure that the tuning exercise has improved speed.

  • Target one bottleneck at a time. The application running characteristics can change after each alteration.

  • Improve a CPU limitation with faster code, better algorithms, and fewer short-lived objects.

  • Improve a system-memory limitation by using fewer objects or smaller long-lived objects.

  • Improve I/O limitations by targeted redesigns or speeding up I/O, perhaps by multithreading the I/O.

• Work with user expectations to provide the appearance of better performance.

  • Hold back releasing tuning improvements until there is at least a 20% improvement in response times.

  • Avoid giving users a false expectation that a task will be finished sooner than it will.

  • Reduce the variation in response times. Bear in mind that users perceive the mean response time as the actual 90th percentile value of the response times.

  • Keep the user interface responsive at all times.

  • Aim to always give user feedback. The interface should not be dead for more than two seconds when carrying out tasks.

  • Provide the ability to abort or carry on alternative tasks.

  • Provide user-selectable tuning parameters where this makes sense.

  • Use threads to separate potentially blocking functions.

  • Calculate "look-ahead" possibilities while the user response is awaited.

  • Provide partial data for viewing as soon as possible, without waiting for all requested data to be received.

  • Cache locally items that may be looked at again or recalculated.

• Quality-test the application after any optimizations have been made.

• Document optimizations fully in the code. Retain old code in comments.