10.5 Timing Multithreaded Tests

I measured timings of the three Counter classes in the previous section using another class, CounterTest. This timing class illustrates some pitfalls you need to avoid when timing multithreaded applications, so I'll go into a little detail about the CounterTest definition.

The first naive implementation of CounterTest is quite simple. Just create a Thread subclass with the run( ) method running timed tests of the classes you are measuring. You need an extra instance variable for the Counter3 class, so the class can be defined as:

package tuning.threads;
   
public class CounterTest
  extends Thread
{
  //instance variable to specify which thread we are.
  int num;
   
  public CounterTest(int threadnum)
  {
    super(  );
    num = threadnum;
  }
   
  // main forks four threads
  public static void main(String[  ] args)
  {
    int REPEAT = (args.length > 0) ? Integer.parseInt(args[0]) : 10000000;
    for (int i = 0; i < 4; i++)
      (new CounterTest(i)).start(  );
  }
   
  public void run(  )
  {
    Counter1.initialize(0);
    long time = System.currentTimeMillis(  );
    for (int i = REPEAT; i > 0; i--)
      Counter1.addAmount(0, 1);
    System.out.println("Counter1 count: " + Counter1.getAmount(0)
      + " time: " + (System.currentTimeMillis(  )-time));
   
    Counter2.initialize(0);
    time = System.currentTimeMillis(  );
    for (int i = REPEAT; i > 0; i--)
      Counter2.addAmount(0, 1);
    System.out.println("Counter2 count: " + Counter2.getAmount(0)
      + " time: " + (System.currentTimeMillis(  )-time));
   
    Counter3.initialize(this);
    time = System.currentTimeMillis(  );
    for (int i = REPEAT; i > 0; i--)
      Counter3.addAmount(0, 1);
    System.out.println("Counter3 count: " + Counter3.getAmount(0)
      + " time: " + (System.currentTimeMillis(  )-time));
  }
}

Unfortunately, this class has two big problems. First, there is no way of knowing that the four threads are running the same test at the same time. With this implementation, it is perfectly possible that one thread is running the Counter1 test while another has already finished that test and is now running the Counter2 test concurrently. This gives incorrect times for both tests because the CPU is being used by another test while you measure the first test. And the synchronization costs are not measured properly because the intention is to test the synchronization costs of running four threads using the same methods at the same time.

The second problem is with the times being measured. The timings are for each thread running its own threaded update to the Counter class. But we should be measuring the time from the first update in any thread to the last update in any thread.

One way to avoid the first pitfall is to synchronize the tests so that they are not started until all the threads are ready. Then all threads can be started at the same time. The second pitfall can be avoided by setting a global time at the start of the first update, then printing the time difference when the last thread finishes.

The full tuning.threads.CounterTest implementation with the correct handling for measurements can be found, along with all the other classes from this book, at http://www.oreilly.com/catalog/javapt2/.