Repeated and dynamic tests, test order

Usually one method marked @Test is one test. JUnit has two other kinds. A repeated test runs the same method several times — handy when the result may depend on the run number or when you want to make sure a test is stable. A dynamic test is created by your code while the tests are running: a factory method returns a list of tests, and JUnit runs each of them. At the end of the lesson — how to set the order of tests and why you should rarely need it.

@RepeatedTest

Create the file src/test/java/com/example/blog/RepositoryRepeatedTest.java:

// src/test/java/com/example/blog/RepositoryRepeatedTest.java
package com.example.blog;
 
import org.junit.jupiter.api.RepeatedTest;
import org.junit.jupiter.api.RepetitionInfo;
 
import static org.junit.jupiter.api.Assertions.assertEquals;
 
class RepositoryRepeatedTest {
 
    private final InMemoryPostRepository repository = new InMemoryPostRepository();
 
    @RepeatedTest(value = 3, name = "{displayName} {currentRepetition}/{totalRepetitions}")
    void idsGrowOneByOne(RepetitionInfo info) {
        int count = info.getCurrentRepetition();
 
        Post last = null;
        for (int i = 0; i < count; i++) {
            last = repository.save(new Post(null, "Post " + i, "anna", false));
        }
 
        assertEquals(count, last.id());
        assertEquals(count, repository.findAll().size());
    }
}

@RepeatedTest(3) runs the method three times. Every repetition is a separate test with a fresh instance of the class, so the repository is empty each time. The RepetitionInfo parameter tells which repetition is running (getCurrentRepetition(), from 1) and how many there are (getTotalRepetitions()). The test saves one, two and then three posts and checks that the ids go 1, 2, 3. In the IDE the runs are named idsGrowOneByOne(RepetitionInfo) 1/3, 2/3, 3/3 thanks to the name template.

[INFO] Running com.example.blog.RepositoryRepeatedTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.030 s -- in com.example.blog.RepositoryRepeatedTest

RepetitionInfo works only in a repeated test. In a plain @Test JUnit has nothing to pass: ParameterResolutionException: No ParameterResolver registered for parameter [org.junit.jupiter.api.RepetitionInfo arg0].

Dynamic tests: @TestFactory

A parameterized test gets its data from annotations or a method. A dynamic test goes further: the tests themselves, with their names, are built by code — for example, from a list of examples or from files in a folder.

// src/test/java/com/example/blog/SlugsDynamicTest.java
package com.example.blog;
 
import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.TestFactory;
 
import java.util.List;
import java.util.stream.Stream;
 
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.DynamicTest.dynamicTest;
 
class SlugsDynamicTest {
 
    record Example(String title, String slug) {
    }
 
    private final List<Example> examples = List.of(
            new Example("Hello", "hello"),
            new Example("Hello, World!", "hello-world"),
            new Example("JUnit & Mockito", "junit-mockito")
    );
 
    @TestFactory
    Stream<DynamicTest> slugs() {
        return examples.stream()
                .map(example -> dynamicTest(
                        "slug of '" + example.title() + "'",
                        () -> assertEquals(example.slug(), Slugs.of(example.title()))
                ));
    }
}

The factory method is marked @TestFactory and returns a Stream, a Collection or another set of DynamicTest. dynamicTest(name, code) makes one test: a display name and a lambda with the checks. Maven counts three tests:

[INFO] Running com.example.blog.SlugsDynamicTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.025 s -- in com.example.blog.SlugsDynamicTest

If one example fails — say, the expected slug is hello-world! — Surefire shows its number, and the IDE shows its name slug of 'Hello, World!':

[ERROR] com.example.blog.SlugsDynamicTest.slugs()[2] -- Time elapsed: 0.007 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <hello-world!> but was: <hello-world>

Note that @BeforeEach and @AfterEach run once around the whole factory method, not around each dynamic test. When the data fits into @CsvSource or @MethodSource, a parameterized test is simpler; take @TestFactory when the set of tests is computed.

Test order

By default JUnit runs the methods of a class in an order that is fixed but deliberately not obvious. If you need a given order, set it on the class:

// src/test/java/com/example/blog/PostStepsTest.java
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
class PostStepsTest {
 
    private final PostService service =
            new PostService(new InMemoryPostRepository(), (to, message) -> { });
 
    @Test
    @Order(1)
    void create() {
        assertEquals("Hello", service.create("  Hello ", "anna").title());
    }
 
    @Test
    @Order(2)
    void publish() {
        Post post = service.create("Hello", "anna");
 
        assertTrue(service.publish(post.id()).published());
    }
}

Tests now run by the number in @Order: create, then publish. Other orderers: MethodOrderer.MethodName (alphabetical), DisplayName, Random.

Note that publish() creates its own post instead of using the one from create(). This is the main rule: the order may make the report nicer, but a test must not depend on it. Each test prepares its own data — otherwise running one test alone or in parallel breaks it (see the lesson "What makes a good test"). The order is really useful only in integration tests with slow steps, and even there it is better avoided.

Common mistake

A factory method that returns nothing. The code inside creates a test, but nobody gives it to JUnit:

@TestFactory
void slugs() {
    dynamicTest("slug of 'Hello'", () -> assertEquals("hello", Slugs.of("Hello")));
}

The build is green, but nothing was checked:

WARNING: TestEngine with ID 'junit-jupiter' encountered a non-critical issue during test discovery:
 
(1) [WARNING] @TestFactory method 'void com.example.blog.VoidFactoryTest.slugs()' must return a single org.junit.jupiter.api.DynamicNode or a Stream, Collection, Iterable, Iterator, Iterator provider, or array of org.junit.jupiter.api.DynamicNode. It will not be executed.
...
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

Watch for Tests run: 0 and discovery warnings. A @TestFactory method must return the tests: Stream<DynamicTest>, List<DynamicTest> or a single DynamicTest.

Thanks for reading!

@RepeatedTest

Dynamic tests: @TestFactory

Test order

Common mistake

Thanks for reading!