JUnit 4.11 - What's new? Test execution order

JUnit 4.11 is a major release of the popular testing framework. One of the problems addressed was a problem introduced by Java 7, more specifically the JVM. In Java 7, you can no longer guarantee that the order of methods returned by reflection is the same as the order in which they are declared in the source file. Actually, this was never guaranteed, but most JVMs did return the methods in source file order. And all of them did return them in a consistent order. With Java 7, even this is no longer guaranteed. The order that the methods are returned can vary from run to run. To quote the javadoc for Class#getMethods(). my emphasis:

Returns an array containing Method objects reflecting all the public member methods of the class or interface represented by this Class object, including those declared by the class or interface and those inherited from superclasses and superinterfaces. Array classes return all the (public) member methods inherited from the Object class. The elements in the array returned are not sorted and are not in any particular order.

So why did JUnit care about this? JUnit finds the tests that it runs using reflection. And the tests are run in this order. So, if a test suite has implicit or explicit dependencies between tests, a test run can sometimes succeed and other times fail.

So, using the following test case as an example:

public class ExecutionOrderTest {
  @Test public void firstTest() { System.out.println("firstTest"); }
  @Test public void secondTest()  { System.out.println("secondTest"); }
  @Test public void thirdTest()  { System.out.println("thirdTest"); }

  public static void main(String[] args) {
    JUnitCore.runClasses(ExecutionOrderTest.class);
  }
}

Using java 1.6 & 4.10, we get:

firstTest
secondTest
thirdTest

Whereas with java 1.7 we get:

thirdTest
firstTest
secondTest

So the order is different. So, what’s the fix? After a lot of discussion (see Sort test methods for predictability), it was decided to make the sort order of methods deterministic, but unpredictable. So, we still get the tests in a strange order, but at least the next time we run the tests, we’ll still get the same order, which makes debugging a lot easier.

However, even with this, there is still a problem. The algorithm used to calculate the deterministic order is based on the hashCode of the method name, it’s pretty obscure. This means that if I have a problem with ordering then I can’t easily fix it. For instance, the hashCode of “secondTest” is 423863078 and “thirdTest” is -585354599. Which means that thirdTest will be executed before secondTest. But if I want for whatever reason to execute thirdTest after secondTest, I have to rename thirdTest to something with a hashCode of greater than 423863078. Yuck. But, there is a solution.

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class ExecutionOrderTest {
  @Test public void firstTest() { System.out.println("firstTest"); }
  @Test public void secondTest()  { System.out.println("secondTest"); }
  @Test public void thirdTest()  { System.out.println("thirdTest"); }

  public static void main(String[] args) {
    JUnitCore.runClasses(ExecutionOrderTest.class);
  }
}

@FixMethodOrder allows the developer to specify that in this case, please execute the tests in order of name ascending. See SortMethodsWith allows the user to choose the order of execution of the methods within a test class. So this will at least allow me to fix the order of my broken (see below) tests. There are three possible values we can specify:

• MethodSorters.JVM: the order in which the methods are returned by the JVM, potentially different order each time
• MethodSorters.DEFAULT: deterministic ordering based upon the hashCode
• MethodSorters.NAME_ASCENDING: order based upon the lexicographic ordering of the names of the tests.

By broken, I mean you shouldn’t have dependencies between tests under normal circumstances. The FixMethodOrder will at least allow you to ‘fix’ your tests until you can work out where the dependency is and eliminate it.