Home
  • Apr 09, 2021
  • Post Nº 165

JUnit 5: Dynamic Tests with TestFactory

A new programming model introduced in JUnit 5.

Introduction

In this article, I am going to share with you how to write dynamic tests in JUnit 5 using @TestFactory. Dynamic testing is a new programming model introduced in JUnit 5. It is useful to create tests that cannot be defined at compile time (e.g. loaded via an external resource) or to create tests that cannot be expressed easily via @ParameterizedTest.

After reading this article, you will understand:

  • The basic syntax of dynamic test
  • Different return types for @TestFactory
  • The lifecycle of dynamic test
  • Checking the result in IDE
  • How to go further from here

Now, let’s get started!

Basic Syntax

According to the JUnit 5 User Guide, the standard @Test annotation in JUnit Jupiter described in Annotations is very similar to the @Test annotation in JUnit 4. Both describe methods that implement test cases. These test cases are static in the sense that they are fully specified at compile-time, and their behavior cannot be changed by anything happening at runtime. Assumptions provide a basic form of dynamic behavior but are intentionally rather limited in their expressiveness.

In addition to these standard tests, a completely new kind of test programming model has been introduced in JUnit Jupiter. This new kind of test is a dynamic test which is generated at runtime by a factory method that is annotated with @TestFactory.

The basic syntax of a dynamic test is:

@TestFactory  // 1
Stream<DynamicTest> dynamicTestStream() {  // 2
  return IntStream.of(0, 3, 6, 9)
      .mapToObj(v ->
          dynamicTest(v + " is a multiple of 3", () -> assertEquals(0, v % 3))  // 3
      );
}

In the example above, we specified several things:

  1. The annotation @TestFactory so that JUnit 5 can recognize this method as a test factory containing multiple dynamic tests.
  2. The return type of the method is a stream of DynamicTest. Note that you don’t have to use Stream and there are other choices. We will talk about that in the next section.
  3. Use static method org.junit.jupiter.api.DynamicTest.dynamicTest to create a dynamic test. Each dynamicTest consists of two parts: a string for the display name and an Executable for the assertions.

Once we have them, we can run the test. There is no additional dependency required. In the following sections, we are going to explore a bit more into detail different pieces.

Return Types For Test Factory

In the section above, we saw that we can return a list of dynamic tests as Stream<DynamicTest>:

Stream<DynamicTest> dynamicTests() { ... }

But it does not have to be in this way. We can also specify other types for returning the tests. As far as this type is iterable, JUnit 5 is happy about it. For example, you can use Collection, Iterable, Iterator, Stream, or array of DynamicTest. Besides that, you can also consider using DynamicContainer` which can contain multiple tests inside it. See JUnit 5 - User Guide for more information about that.

Lifecycle Of Dynamic Tests

The execution lifecycle of a dynamic test is different from the standard @Test case. The lifecycle callbacks, @BeforeEach and @AfterEach, are not executed for each dynamic test, but the whole @TestFactory. Therefore, we need to be very careful about using class-level variables because they won’t be reset properly.

Stardard @Test:

  1. Execute @BeforeEach
  2. Execute @Test
  3. Execute @AfterEach

Dynamic tests via @TestFactory:

  1. Execute @BeforeEach
  2. Execute @TestFactory
    • Execute dynamic test 1
    • Execute dynamic test 2
    • Execute dynamic test 3
    • …
  3. Execute @AfterEach

IDE

When running dynamic tests in IntelliJ IDEA (2020.3.1), you can find the results as follows:

Dynamic tests in IntelliJ IDEA

where each dynamic test has its result and its display name.

Should We Use Dynamic Tests?

Actually, I prefer using @ParameterizedTest over @TestFactory because it has the full lifecycle support (@BeforeEach and @AfterEach) while the test factory doesn’t. Both of them support display names so it’s not a problem. Usually, the test cases can be defined at compile-time.

So why should we dynamic tests?

I believe there are two reasons: when the tests cannot be expressed at compile-time or when the parameterized tests are not good enough.

  1. Runtime test sources. When the tests cannot be expressed at compiled time, you may want to load them at runtime. Dynamic tests support this via the following methods:

    DynamicTest.dynamicTest(String, URI, Executable)
DynamicContainer.dynamicContainer(String, URI, Stream)

    Therefore, you can pass the sources via a URI. It can be something in the classpath, in the filesystem, etc.

  2. Because parameterized tests are not good enough. This is my test. I wanted to use some exceptions as input sources, and assert the exception handling mechanism by asserting these exceptions one after another. However, @ParameterizedTest seems only support basic Java types: primitives or String, so passing an exception as input is not possible. So I ended up using dynamic tests for this purpose.

There are probably other motivations as well. Please let me know what you think by leaving a comment :)

Going Further

How to go further from here?

You can also see the source code of this article on GitHub under mincong-h/java-examples.

Conclusion

In this article, we saw how to write dynamic tests via @TestFactory, the different return types for @TestFactory, such as Collection, Iterable, Iterator, array, or Stream of dynamic tests. We also see the lifecycle of dynamic tests, which do not benefit from the @BeforeEach and @AfterEach callbacks for each individual dynamic test, how do the results look like in IDE, and how to go further from here. Interested to know more? You can subscribe to the feed of my blog, follow me on Twitter or GitHub. Hope you enjoy this article, see you the next time!

References

Search