Running tests and reading the results

A test is only useful if you run it often and understand what it says. This lesson covers how to run all tests or just some of them from the terminal and how to read Maven's report — both when everything passes and when something breaks.

All tests

From the project folder (where pom.xml is):

mvn test

Maven compiles the code and the tests, and Surefire runs every test class:

[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.example.shop.PriceCalculatorTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 s -- in com.example.shop.PriceCalculatorTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] BUILD SUCCESS

Counter	Meaning
`Tests run`	How many tests were executed
`Failures`	An assertion did not hold: the result differs from the expected one
`Errors`	The test crashed with an unexpected exception, for example a `NullPointerException`
`Skipped`	The test was disabled or skipped by an assumption

Part of the tests

mvn test -Dtest=PriceCalculatorTest              # one class
mvn test -Dtest='PriceCalculatorTest#discount*'  # methods by a pattern
mvn test -Dtest='Price*'                         # classes by a pattern

The quotes protect # and * from the shell. In IntelliJ IDEA the same is done by the green arrow next to a class or a method.

A failing test

Suppose the test author miscalculated and expects $40.00 after a 10% discount on $50.00. Let us also add a test that calls total() with the quantity -1 and does not expect an exception:

// src/test/java/com/example/shop/PriceCalculatorTest.java
@Test
void negativeQuantity() {
    PriceCalculator calculator = new PriceCalculator();
 
    calculator.total(new BigDecimal("12.00"), -1);
}

The output of mvn test (stack frames of JUnit itself are removed):

[ERROR] Tests run: 3, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 0.150 s <<< FAILURE! -- in com.example.shop.PriceCalculatorTest
[ERROR] com.example.shop.PriceCalculatorTest.negativeQuantity -- Time elapsed: 0.085 s <<< ERROR!
java.lang.IllegalArgumentException: Quantity must be positive: -1
	at com.example.shop.PriceCalculator.total(PriceCalculator.java:10)
	at com.example.shop.PriceCalculatorTest.negativeQuantity(PriceCalculatorTest.java:33)
 
[ERROR] com.example.shop.PriceCalculatorTest.discountReducesAmount -- Time elapsed: 0.009 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <40.00> but was: <45.00>
	at com.example.shop.PriceCalculatorTest.discountReducesAmount(PriceCalculatorTest.java:26)
 
[INFO] Results:
[ERROR] Failures:
[ERROR]   PriceCalculatorTest.discountReducesAmount:26 expected: <40.00> but was: <45.00>
[ERROR] Errors:
[ERROR]   PriceCalculatorTest.negativeQuantity:33 » IllegalArgument Quantity must be positive: -1
[ERROR] Tests run: 3, Failures: 1, Errors: 1, Skipped: 0
[INFO] BUILD FAILURE

Read it from the top: the class and method, FAILURE or ERROR, the message and the line in the test file. A failure means expected ... but was ...: decide who is wrong, the code or the test. Here it is the test: 10% of $50.00 is $5.00, so the result is $45.00. An error means the code threw an exception the test did not expect; the stack trace shows where. How to test exceptions on purpose is covered in the next chapter.

Full reports for each class are saved to target/surefire-reports: a .txt file with the same text and an .xml file that CI servers read.

Which classes are tests

By default Surefire runs classes whose names match these patterns:

Pattern	Example
`*/Test.java`	`TestPriceCalculator`
`*/Test.java`	`PriceCalculatorTest`
`*/Tests.java`	`PriceCalculatorTests`
`*/TestCase.java`	`PriceCalculatorTestCase`

A class named PriceCalculatorCheck compiles, but Maven never runs it. The IDE does run it, so the mistake goes unnoticed until CI.

Common mistake

A typo in -Dtest stops the build:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.6.0:test (default-test) on project blog: No tests matching pattern "PriceCalculatorTests" were executed! (Set -Dsurefire.failIfNoSpecifiedTests=false to ignore this error.) -> [Help 1]

The class is called PriceCalculatorTest, without the s. Check the name: the pattern must match the simple name of the class.

Thanks for reading!