converting 2K+ tests to junit 5 in one day

I set myself a goal to see if I could convert all the CodeRanch JForum tests to JUnit 5 in one day. And I mean with Java 5 package names, not just running them in JUnit 4 mode. (I wanted to see what edge cases I encountered). Let’s see how that went.

Forking

Ant doesn’t currently support JUnit 5. So I decided to create a fork of the code base for playing with JUnit 5. Which is fine. I wanted to automate as much of the process as possible so it’ll be easy to redo when the time comes. It’s also good because then I don’t have to worry about whether everyone’s IDE supports JUnit 5 yet. IntelliJ does if you are on the latest version and Eclipse does if you fiddle with it. Probably better to wait until after the official release of JUnit 5 and Eclipse before using JUnit 5 in the trunk anyway.

Scope reduction

I decided to only update the unit tests. We also have some functional tests with an odd hacked together way of loading the database.  (I wrote a lot of it so not criticizing others.) I’m sure this can be improved in JUnit 5. Will wait until release for that as well.

JUnit 3.8

We have 15 classes that still reference JUnit 3.8. They used a mix of a superclass that did inherit from JUnit’s test case and some that used JUnit 4 style annotations too. I decided to go through these by hand in the trunk (and my branch) and get rid of the references to JUnit 3.8. This was a pain because each one was a special snowflake. One even said it was deprecated with replacement code listed. But it took under an hour; a few minutes each. So not terrible.

As I was going, I missed noticing that the class missed annotation. I was excited to see that SonarLint flags this!

Migrating most of the code

I ran a program to migrate most of the code to the new syntax. Here’s the project on github.

This left me with 83 compiler errors to look at. Of those…

Manual clean up

  • Parameterized tests. The manual gives good examples of the options. Conversion was easy; I used a method source. I did run into two problems.
    • SonarLint false positive – If you only have parameterized tests, SonarLint still flags the class. This was fixed in Sonar per SONARJAVA-2390. However, SonarLint hasn’t been released since so need to ignore the error in my mind.
    • Eclipse false positive – If you only have parameterized tests, Eclipse doesn’t recognize it as being a JUnit test. I didn’t search to see if this was reported since it is so easy to work around. To “solve” both these problems, I added a non-parameteirzed test to the class.
  • Timeout. I had one test that used the @Test(timeout) parameter. It was clear from the manual how to convert this. I like that specifying the time uses Java 8’s Duration class. This lets you specify the timeout in a readable way rather than in milliseconds.
  • Expected exception. I had 16 tests that used the @Test(expected) parameter. I thought about automating this but decided against it. This is a good opportunity to add an assertion about the message to most of them. Which we should have done originally. But it was *so* easy to just write the type and be done with it before. Also, it isn’t 16 distinct classes. A number of them have multiple validation type methods.
  • About 25 of our tests use JMock. @RunWith(JMock.class) doesn’t go with JUnit 5. Since JMock was abandoned (last release in 2012), I decided to switch to the “long way” and call context.assertIsSatisfied(); for now. About half of them extend the same superclass at least. I also made a note that we should migrate away from JMock.

Then I attempted to run all the converted tests

I ran into one problem

  • One of our tests was relying on the order in which the @Before methods were run in the superclass. This behavior was never guaranteed and changed from JUnit 4 to JUnit 5. I fixed the code to not rely on the order. (And or course, the place this happened was a superclass test so it resulted in a large number of failures until I figured out what was going on!)

Observations

I met my challenge. 2K+ tests migrated in one day. And I observed that one superclass caused the vast majority of the special cases!

junit 5 and intellij

I use Eclipse at home for development. And mostly Maven (except for two projects that use Ant because some of the people working on them have less reliable internet connections).

After preparing to run JUnit 5 with Eclipse and Maven, I wanted to check the impact on IntelliJ. It’s been a long time since I’ve tried using IntelliJ. The internet says it supports JUnit 5 so I wanted to try it to confirm. I need to know this:

  1. So I know what IDEs people can use at my JavaOne JUnit 5 hands on lab
  2. So I know whether upgrading the CodeRanch JForum fork to JUnit 5 will be a problem for the developers using IntelliJ

Installing IntelliJ

  1. Download Mac version of free community edition from JetBrains
  2. Drag .dmg folder to Applications folder
  3. I chose not import settings. I don’t even remember the last time I ran IntelliJ on my Mac so those settings aren’t likely to be useful.
  4. After accepting the privacy policy, IntelliJ opens and I chose all the defaults.

Running the project

To run the project I migrated, I chose:

  • Check out from version control > git
  • https://github.com/boyarsky/oracle-java-cert-objectives-history.git and clone
  • IntelliJ asked if wanted to create a project based on the pom and I said yes
  • Choose command 1 for project view
  • Tried to open a Java class and got prompted to configure the SDK. I chose the latest Java 8 on my machine. No need to use Java 9 for what I’m doing. /Library/Java/JavaVirtualMachines/jdk1.8.0_xxx.jdk/Contents/Home
  • Build using build pulldown
  • Run using run pulldown

The result

IntelliJ was able to recognize the JUnit 5 annotations/methods and run them. It’s a bit ahead of Eclipse in this space so JUnit 5 won’t be a problem if they are using the latest version of IntelliJ.

upgrading to junit 5 parameterized tests for the first time

I’ve been playing with JUnit 5 for a while. Migrating existing project is something else so I decided to try with my small project cert objectives checker. This project does run on Jenkins so is a good test because it isn’t just on my machine. First I set up Eclipse and Maven to work with JUnit 5.

Switch to Jupiter APIs

Since I only had 5 classes, I updated by hand. For each class:

  1. Changed imports to
    import static org.junit.jupiter.api.Assertions.*;
    import static org.hamcrest.MatcherAssert.assertThat;
    import org.junit.jupiter.api.*;

    (The hamcrest one is so I can keep using assertThat)

  2. Changed assertEquals/assertTrue/etc calls to have String message as last parameter instead of first.
  3. Changed @Before to @BeforeEach
  4. Got distracted and updated tests to Java 8 syntax (nothing to do with JUnit)
  5. Changed to use new ParameterizedTest syntax. More on this…

Parameterized tests

Interestingly Eclipse knows about the new Parameterized Tests dependency natively. But if you use Maven, you need to add a dependency:

<dependency>
  <groupId>org.junit.jupiter</groupId>
  <artifactId>junit-jupiter-params</artifactId>
  <version>5.0.0-RC2</version>
</dependency>

Now let’s compare the before and after. I’ve omitted parts that are the same in both (aka the detailed logic of my test). Not that it is secret of anything. (After all the project is public on github.) But it makes it easier to compare the differences with less code.

JUnit 4 version

import static org.junit.Assert.*;

import java.io.*;
import java.net.*;
import java.nio.file.*;
import java.util.*;

import javax.xml.parsers.*;

import org.junit.*;
import org.junit.runner.*;
import org.junit.runners.*;
import org.junit.runners.Parameterized.Parameters;
import org.w3c.dom.*;

@RunWith(Parameterized.class)
public class CheckForChangesIT {

  @Parameters
  public static List<CertsToCheckEnum[]> suite() {
    List<CertsToCheckEnum[]> result = new ArrayList<>();
    for (CertsToCheckEnum element : CertsToCheckEnum.values()) {
       result.add(new CertsToCheckEnum[] { element });
    }
    return result;
  }
  // ----------------------------------------------------
  private CertsToCheckEnum certToCheck;
  private InputStream stream;
  private Document doc;

  public CheckForChangesIT(CertsToCheckEnum certToCheck) {
    this.certToCheck = certToCheck;
  }
  // ----------------------------------------------------
  @Before
  public void connect() throws Exception {
    String url = XML_URL + certToCheck.getExamNumber();
    stream = new URL(url).openStream();
  }
  @After
  public void close() {
    if (stream != null) {
      try {
        stream.close();
      } catch (Exception e) {
        // ignore
      }
    }
  }
  // ----------------------------------------------------
  @Test
  public void upToDate() throws Exception {
    parseDocument();
    String currentData = convertToString();
    assertSameAsExisting(currentData);
  }

JUnit 5 version

import static org.junit.jupiter.api.Assertions.*;

import java.io.*;
import java.net.*;
import java.nio.file.*;

import javax.xml.parsers.*;

import org.junit.jupiter.params.*;
import org.junit.jupiter.params.provider.*;
import org.w3c.dom.*;

public class CheckForChangesIT {

  private CertsToCheckEnum certToCheck;
  private Document doc;
  // ----------------------------------------------------
  @ParameterizedTest
  @EnumSource(CertsToCheckEnum.class)
  public void upToDate(CertsToCheckEnum c) throws Exception {
    certToCheck = c;
    String url = XML_URL + certToCheck.getExamNumber();
    try (InputStream stream = new URL(url).openStream()) {
      parseDocument(stream);
      String currentData = convertToString();
      assertSameAsExisting(currentData);
    }
  }

Key differences between JUnit 4 and 5 Parameterized Test

  • The JUnit 5 annotation has a different name and is placed on the test rather than the class. This is good because it lets you have some parameterized tests in a class and some regular tests. With JUnit 4, I had to arbitrarily create another class for the non-parameterized parts.
  • JUnit 5 has a number of annotations for building the parameter list without having to write custom code.
  • JUnit 5 injects the parameter as a method parameter rather than an instance variable.
    • In many ways this is nice. For example, I am now using a try-with-resources to close the stream rather than creating in @BeforeEach and having ugly logic to clean it up later. I could have done this in JUnit 4 but it seemed less natural.
    • In the case of my “legacy” test, I use the previously injected instance variable all over the class.  Hence my hack of storing the local variable in the original instance variable as the first line of my test method.
  • The JUnit 5 version is a good bit shorter

One problem

I encountered a minor Eclipse bug with parameterized tests. So I reported it.