JUnit 4.11 - What's new? Hamcrest 1.3

JUnit 4.11 is a major release of the popular testing framework. JUnit now supports version 1.3 of Hamcrest. Hamcrest is a wonderful library to allow matcher objects for match rules to be defined declaratively. JUnit pre 4.11 uses these matchers, and indeed includes version 1.1 of the hamcrest libraries. Which is a problem because hamcrest is now version 1.3, and some of the method signatures have changed.

Let’s have an example. Using the following test as an example:

import static org.hamcrest.collection.IsCollectionWithSize.hasSize;

public class HamcrestTest {
  @Test
  public void test() {
    List<String> list = Arrays.asList("one", "two");

 // test that the collection has a size of 1
    Assert.assertThat(list, hasSize(1));
  }
}

If we compile this and run with JUnit 4.10 and hamcrest-all-1.3.jar.

$ java -cp "hamcrest-all-1.3.jar:junit-4.10.jar:." org.junit.runner.JUnitCore hamcrest.HamcrestTest

and we get

There was 1 failure:
1) test(hamcrest.HamcrestTest)
java.lang.AssertionError: 
Expected: a collection with size <1>
 got: <[one, two]>

This seems to work, but there are two things to note here. The first is that the message is wrong. It’s actually using a different matcher. It should be:

Expected: a collection with size <1>
 but: collection size was <2>

The real problem becomes obvious when we swap the order of the classpath, and put junit before hamcrest.

$ java -cp "junit-4.10.jar:hamcrest-all-1.3.jar:." org.junit.runner.JUnitCore hamcrest.HamcrestTest

There was 1 failure:
1) test(hamcrest.HamcrestTest)
 java.lang.NoSuchMethodError: org.hamcrest.Matcher.describeMismatch(Ljava/lang/Object;Lorg/hamcrest/Description;)V

The problem is that JUnit 4.10 includes some of the org.hamcrest 1.1 class files, so if you mix and match a class from the junit jar (version 1.1) and the hamcrest jar (1.3), then, unsurprisingly, it doesn’t work. This is not good.

So, to the fix. JUnit 4.11 no longer includes the org.hamcrest classes. All of the methods in org.junit.matcher.JUnitMatchers have been deprecated: just replace them with the equivalent from org.hamcrest.CoreMatchers. Use the hamcrest matchers directly. junit-dep is now also deprecated, because hamcrest is now a direct dependency.

So, if we run the above with junit 4.11, we get the correct error message:

$ java -cp "junit-4.11.jar:hamcrest-all-1.3.jar:." org.junit.runner.JUnitCore hamcrest.HamcrestTest

1) test(hamcrest.HamcrestTest)
java.lang.AssertionError: 
Expected: a collection with size <1>
 but: collection size was <2>

JUnit 4.11 - What's new? Test execution order

JUnit 4.11 is a major release of the popular testing framework. One of the problems addressed was a problem introduced by Java 7, more specifically the JVM. In Java 7, you can no longer guarantee that the order of methods returned by reflection is the same as the order in which they are declared in the source file. Actually, this was never guaranteed, but most JVMs did return the methods in source file order. And all of them did return them in a consistent order. With Java 7, even this is no longer guaranteed. The order that the methods are returned can vary from run to run. To quote the javadoc for Class#getMethods(). my emphasis:

Returns an array containing Method objects reflecting all the public member methods of the class or interface represented by this Class object, including those declared by the class or interface and those inherited from superclasses and superinterfaces. Array classes return all the (public) member methods inherited from the Object class. The elements in the array returned are not sorted and are not in any particular order.

So why did JUnit care about this? JUnit finds the tests that it runs using reflection. And the tests are run in this order. So, if a test suite has implicit or explicit dependencies between tests, a test run can sometimes succeed and other times fail.

So, using the following test case as an example:

public class ExecutionOrderTest {
  @Test public void firstTest() { System.out.println("firstTest"); }
  @Test public void secondTest()  { System.out.println("secondTest"); }
  @Test public void thirdTest()  { System.out.println("thirdTest"); }

  public static void main(String[] args) {
    JUnitCore.runClasses(ExecutionOrderTest.class);
  }
}

Using java 1.6 & 4.10, we get:

firstTest
secondTest
thirdTest

Whereas with java 1.7 we get:

thirdTest
firstTest
secondTest

So the order is different. So, what’s the fix? After a lot of discussion (see Sort test methods for predictability), it was decided to make the sort order of methods deterministic, but unpredictable. So, we still get the tests in a strange order, but at least the next time we run the tests, we’ll still get the same order, which makes debugging a lot easier.

However, even with this, there is still a problem. The algorithm used to calculate the deterministic order is based on the hashCode of the method name, it’s pretty obscure. This means that if I have a problem with ordering then I can’t easily fix it. For instance, the hashCode of “secondTest” is 423863078 and “thirdTest” is -585354599. Which means that thirdTest will be executed before secondTest. But if I want for whatever reason to execute thirdTest after secondTest, I have to rename thirdTest to something with a hashCode of greater than 423863078. Yuck. But, there is a solution.

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class ExecutionOrderTest {
  @Test public void firstTest() { System.out.println("firstTest"); }
  @Test public void secondTest()  { System.out.println("secondTest"); }
  @Test public void thirdTest()  { System.out.println("thirdTest"); }

  public static void main(String[] args) {
    JUnitCore.runClasses(ExecutionOrderTest.class);
  }
}

@FixMethodOrder allows the developer to specify that in this case, please execute the tests in order of name ascending. See SortMethodsWith allows the user to choose the order of execution of the methods within a test class. So this will at least allow me to fix the order of my broken (see below) tests. There are three possible values we can specify:

• MethodSorters.JVM: the order in which the methods are returned by the JVM, potentially different order each time
• MethodSorters.DEFAULT: deterministic ordering based upon the hashCode
• MethodSorters.NAME_ASCENDING: order based upon the lexicographic ordering of the names of the tests.

By broken, I mean you shouldn’t have dependencies between tests under normal circumstances. The FixMethodOrder will at least allow you to ‘fix’ your tests until you can work out where the dependency is and eliminate it.

JUnit 4.11 - What's new? Parameterized descriptions

JUnit 4.11 is a major release of the popular testing framework. Amongst the changes were changes to how Parameterized tests are described by the various runners (maven surefire, Eclipse etc).

With JUnit 4.10, when you ran a Parameterized test, all you got in the description was an integer index, which wasn’t very useful. Taking the following test class as an example:

@RunWith(Parameterized.class)
public class ParameterizedTest {
  @Parameters
  public static Iterable<Object[]> data() {
    return Arrays.asList(new Object[][] {
      { 0, 0 },
      { 1, 1 },
      { 2, 1 },
    });
  }

  private int input;
  private int expected;

  public ParameterizedTest(int input, int expected) {
    this.input = input;
    this.expected = expected;
  }

  @Test public void test() {
    assertEquals(expected, Fibonacci.compute(input));
  }

  public static void main(String[] args) {
    RunListener runListener = new RunListener() {
      public void testFailure(Failure failure) throws Exception {
        System.out.println("testFailure " + failure.getDescription().getDisplayName());
      }
    };
    JUnitCore jUnitCore = new JUnitCore();
    jUnitCore.addListener(runListener);
    jUnitCore.run(ParameterizedTest.class);
  }
}

When we run this (as a main method), we get the following output:

testFailure test[0](uk.co.farwell.ParameterizedTest)
testFailure test[1](uk.co.farwell.ParameterizedTest)
testFailure test[2](uk.co.farwell.ParameterizedTest)

The output isn’t very useful when we’re debugging tests, especially if there are lots of them. So, JUnit 4.11 allows the description of the test to be modified by adding a annotation parameter:

@Parameters(name = "{index}: fib({0}) should be {1}")
public static Iterable<Object[]> data() {
  return Arrays.asList(new Object[][] {
    { 0, 0 },
    { 1, 1 },
    { 2, 1 },
  });
}

The {0} is replaced by the first parameter for the test, and so on. Now the description is much more, well, descriptive:

testFailure test[0: fib(0) should be 0](uk.co.farwell.ParameterizedTest)
testFailure test[1: fib(1) should be 1](uk.co.farwell.ParameterizedTest)
testFailure test[2: fib(2) should be 1](uk.co.farwell.ParameterizedTest)

However, when running with the junit runner in Eclipse, you need to be careful. When it creates the descriptions for the test classes, it uses (approximately) ‘uk.co.farwell.ParameterizedTest(test)’. It then uses the name of the test to work out which class and method to jump to when you double click on the test in the junit view. So if, as above, we use parentheses in the description, this doesn’t work. It also screws up the display of the test name in the junit view.

The moral of the story: Don’t use parentheses in your descriptions for parameterized tests.

JUnit 4.11 - What's new? Rules

JUnit 4.11 is a major release of the popular testing framework. Amongst the things I contributed were changes to @Rule and @ClassRule, to allow these annotations to be used on a method as well as a public field. I wanted this because I use Rules heavily in Java, and I wanted to be able to use them in Scala as well. Mainly to make the transition from Java to Scala easier.

Let’s take an example:

public class JavaRuleTest {
  @Rule public ExpectedException expectedException = ExpectedException.none();
 
  @Test
  public void test() {
    expectedException.expect(NumberFormatException.class);
    Integer.valueOf("foo");
  }
}

So this test uses the ExpectedException rule which allows you to say that you expect to receive an exception, in this case a NumberFormatException. Which we do. So the test passes. If we translate this directly to Scala, we get:

class ScalaRuleTest {
  @Rule val expectedException = ExpectedException.none()
 
  @Test def test() {
    expectedException.expect(classOf[NumberFormatException])
    Integer.valueOf("foo")
  }
}

But this produces an error:

The @Rule 'expectedException' must be public.

Although we’re defining a ‘public’ field expectedException, Scala implements this as a private field with an accessor method. So JUnit expects a public field, but finds a private one; the test fails. But now, with JUnit 4.11, we can apply a @Rule to a method:

class ScalaRuleTest {
  val expectedException = ExpectedException.none()
 
  @Rule def expectedExceptionDef = expectedException
 
  @Test def test() {
    expectedException.expect(classOf[NumberFormatException])
    Integer.valueOf("foo")
  }
}

We still need the val expectedException, so that we can refer to it in test(), but we add expectedExceptionDef and move the @Rule annotation to there. And now the test passes.

Similarly, for @ClassRule, you can do the same, but Scala doesn’t have static variables/methods, so you’ll need to add it to the companion object:

object ScalaClassRuleTest {
  @ClassRule def externalResource = new ExternalResource() {
 protected override def before() = println("before")
 protected override def after() = println("after")
  }
}

class ScalaClassRuleTest {
  @Test def test() {
 System.out.println("a test")
  }
}

This change allows me to take my Java code and translate it (almost) directly into Scala, as a stepping stone to rewriting everything completely to use Scalatest or Specs2. Also, other Java programmers may find it useful :-)

Generating bytecode with errors in Scala

Following this blog post from James Iry: Type errors as warnings, and as it's something I've been thinking about recently applying to the Scala-IDE (Eclipse plugin for Scala),
I thought I'd stick my oar in as well.

The Eclipse JDT compiler creates the .class file even when there are errors in the source code. For instance:

public class Foo {
 public void error() {
  int foo = "bar";
 }

 public void ok() {
  int foo = 4;
 }
}

If you take the above java as an example, the javac compiler would not generate the .class file, but the Eclipse JDT compiler does.
It produces the equivalent of the following (using JAD):

public class Foo {
    public Foo() {}

    public void error() {
        throw new Error("Unresolved compilation problem: \n" +
        "\tType mismatch: cannot convert from String to int\n");
    }

    public void ok() {
        int foo = 4;
    }
}

Note that this still shows up as a compilation error in Eclipse, in the Problems view. Note also that calling the error() method will never work.

So should we apply the same trick to Scala? Yes.

But why is this useful? When you're refactoring, or just plain developing, you don't have to fix all of your errors at once.
If you're doing a spike, you can do a single change, make sure it works by running the unit tests (for that class), and then move on to the next implementation.
You can experiment. You can see if a solution works.

Note that I am not suggesting relaxing any type constraints, or any constraints of any kind.
Note also that I've never seen problems come from this feature of Eclipse either; actually, I've never worked on a project where the Eclipse
compiler was used for producing production code. It's always been javac.
I'm also not suggesting that this be available with the normal scalac compiler.
I think it should only be available from the Eclipse IDE, to stop leakage into production code.