Health checks and metrics

When a service runs on a server or in Kubernetes, something has to ask it regularly: "are you alive? are you ready for requests?". These questions are called health checks. If the service does not answer, the orchestrator restarts it. Metrics are numbers about how the service works: how many requests arrived, how long a response took, how many posts were created. A monitoring system such as Prometheus collects them and draws charts.

Adding them

./mvnw quarkus:add-extension -Dextensions="smallrye-health,micrometer-registry-prometheus"

smallrye-health adds the health endpoints, micrometer-registry-prometheus collects metrics through the Micrometer library and exposes them in the Prometheus format.

Health endpoints

Quarkus adds some checks itself. The database extension registers a connection check — without a single line of code.

Your own check

Let's add a readiness check that reports the number of posts:

// src/main/java/com/example/blog/PostsHealthCheck.java
package com.example.blog;
 
import jakarta.enterprise.context.ApplicationScoped;
import org.eclipse.microprofile.health.HealthCheck;
import org.eclipse.microprofile.health.HealthCheckResponse;
import org.eclipse.microprofile.health.Readiness;
 
@Readiness
@ApplicationScoped
public class PostsHealthCheck implements HealthCheck {
 
    @Override
    public HealthCheckResponse call() {
        long count = Post.count();
        return HealthCheckResponse.named("posts")
                .up()
                .withData("count", count)
                .build();
    }
}

@Readiness makes it a readiness check, @Liveness a liveness check. The call() method returns a name, the up() or down() state and extra data. If Post.count() throws an exception, for example because the database is unreachable, the check counts as failed.

curl http://localhost:8080/q/health

{
    "status": "UP",
    "checks": [
        {
            "name": "posts",
            "status": "UP",
            "data": {
                "count": 2
            }
        },
        {
            "name": "Database connections health check",
            "status": "UP",
            "data": {
                "<default>": "UP"
            }
        }
    ]
}

The overall status is UP only if all checks are UP. The response code is then 200, otherwise 503 Service Unavailable. The orchestrator looks at the code.

Metrics

With Micrometer added, metrics are available at /q/metrics. Quarkus counts HTTP requests itself. Let's add our own counter of created posts:

// src/main/java/com/example/blog/PostResource.java
@Inject
MeterRegistry registry;
 
@POST
@Transactional
public Response create(@Valid NewPost newPost) {
    Post post = new Post();
    post.title = newPost.title();
    post.body = newPost.body();
    post.persist();
    registry.counter("blog.posts.created").increment();
    return Response.created(URI.create("/posts/" + post.id)).entity(post).build();
}

MeterRegistry (import io.micrometer.core.instrument.MeterRegistry) is the registry of metrics. counter(...) finds or creates a counter, increment() adds one to it. Create a post and look at the metrics:

curl -s http://localhost:8080/q/metrics | grep -E "^blog_posts|^http_server_requests_seconds_count"

blog_posts_created_total 1.0
http_server_requests_seconds_count{method="POST",outcome="SUCCESS",status="201",uri="/posts"} 1.0

The name blog.posts.created became blog_posts_created_total in the Prometheus format. The http_server_requests_seconds_count metric counts requests separately by method, address and status. On Windows, instead of grep, find the lines with the browser's search.

Common mistake

Do not check external services, such as the database, in liveness. If the database goes down, liveness becomes DOWN, and the orchestrator starts restarting every copy of the service, although a restart will not fix the database. External dependencies belong in readiness: the service temporarily stops receiving requests but keeps running and recovers by itself. That is why our posts check is marked @Readiness, and /q/health/live has no checks yet — the answer is "checks": [] with the UP status.