A Backend Dev's Performance Tuning in Practice (2): Spring Boot Actuator

Foreword

Narrator: Someone comes to shop for a system.

(A sharp screech of motorcycle brakes)

Your boss: Buddy, how much RPS can this system of yours actually handle?

You: 5000 RPS a day.

Your boss: What’s up… is your API made of gold, or are the third-party modules made of gold?

Caught off guard by that question, you have nothing to say. You wanted to argue that this is just a run-of-the-mill system, so cut it some slack, but your boss isn’t the kind of guy you can bluff your way past either — you need more information. If you don’t actually know how your own system runs, that’s undoubtedly fatal. Beyond system architecture, when it’s time to do performance tuning, you need a way to measure the system’s current running state — and the tool I want to talk about this time is Spring Actuator.

What Is Spring Actuator?

Spring Boot Actuator is the official production-monitoring and management module provided by Spring Boot. Its core value is that just by adding one dependency and doing a bit of configuration, Spring Boot will automatically expose your application’s internal state as a series of directly-accessible HTTP endpoints (or via JMX).

To use it, first pull in the spring-boot-starter-actuator dependency:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

With just the dependency added and nothing configured, Actuator by default only exposes the two most harmless endpoints: /actuator/health and /actuator/info. But if we want detailed observability, we have to manually configure which metrics we want exposed, one by one.

management:
  endpoints:
    web:
      exposure:
        include: health, metrics, env, beans, loggers, threaddump, prometheus
      base-path: /actuator # defaults to /actuator, can be omitted

  endpoint:
    health:
      show-details: always # defaults to never, otherwise /health only returns {"status":"UP"} with no details
    shutdown:
      enabled: false # this one's dangerous, don't turn it on lightly

  server:
    port: 9090 # recommend a dedicated port, separate from your business API, for easier network isolation

  metrics:
    tags:
      application: ${spring.application.name} # gives the metrics Prometheus scrapes some identifiability

Once configured, you’ll have a whole row of endpoints for querying system state. Here are a few of the most commonly used ones — ones you’ll bump into pretty much every day:

One reminder: Actuator isn’t a “the more endpoints exposed, the better” kind of thing. Almost all of these endpoints are sensitive in nature — if leaked, an attacker could get an intimate picture of your system. So the usual approach in production is to put Actuator on its own management.server.port, separate from your business API, and then pair it with a firewall or Spring Security to restrict access to internal networks or specific identities only (we’ll actually do this in a hands-on walkthrough later).

Besides metrics that are just for viewing, Actuator also has a handful of endpoints that let you directly change system behavior. The most classic example is /actuator/loggers — you can temporarily crank up the log level of a given package without restarting the service, then dial it back down once you’re done troubleshooting:

# Query the current log level for the com.example.demo package
curl http://localhost:9090/actuator/loggers/com.example.demo
# Query the current log level for the com.example.demo.DemoController class
curl http://localhost:9090/actuator/loggers/com.example.demo.DemoController

# Dynamically switch it to DEBUG, effective immediately, no restart needed
curl -X POST http://localhost:9090/actuator/loggers/com.example.demo \
  -H "Content-Type: application/json" \
  -d '{"configuredLevel": "DEBUG"}'

This “field emergency response” capability is especially useful when production is throwing errors and you can’t afford to restart the service — flip on DEBUG to see the problem clearly, confirm it, then switch back to INFO, all without a redeploy.

For someone encountering these metrics for the first time, they can feel pretty tangled and complex. We need a simpler way to look at them — which brings us to Prometheus and Grafana.

What Is Prometheus?

Prometheus is an open-source Time Series Database, and also one of the de facto standards for monitoring systems. It operates on a Pull model — rather than your application pushing data to it, Prometheus goes and fetches the current metrics from some endpoint on each service at a fixed interval (e.g. every 15 seconds), storing each fetch as a timestamped record.

And that “some endpoint” happens to be the /actuator/prometheus we mentioned in the table above. But spring-boot-starter-actuator alone isn’t enough — this endpoint doesn’t exist by default. You need to add one more dependency so Micrometer (Spring Boot’s built-in metrics facade) can translate the data into a format Prometheus understands:

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

With that added, hitting /actuator/prometheus will show you a long stream of plain-text data like this:

http_server_requests_seconds_bucket{uri="/api/v1/ticket",le="0.1",} 823.0
http_server_requests_seconds_bucket{uri="/api/v1/ticket",le="0.5",} 910.0
http_server_requests_seconds_count{uri="/api/v1/ticket",} 915.0
http_server_requests_seconds_sum{uri="/api/v1/ticket",} 42.183

Next, you need to tell Prometheus where to fetch this data from, via its own configuration file, prometheus.yml:

scrape_configs:
  - job_name: "ticket-service"
    metrics_path: "/actuator/prometheus"
    scrape_interval: 15s
    static_configs:
      - targets: ["localhost:9090"] # your management.server.port

Once that’s configured and Prometheus is running, it’ll hit /actuator/prometheus every 15 seconds and store the current numbers into its own time series database, letting you use PromQL (Prometheus’s dedicated query language) to do all sorts of after-the-fact analysis — calculating the P95 latency over the last 5 minutes, the QPS of a given API, JVM memory usage trends, and so on.

What Is Grafana?

If Prometheus is responsible for “collecting data, storing it, and letting you query it,” then Grafana is responsible for “turning the queried data into something a human can actually understand.” It doesn’t store data itself — instead, it connects to Prometheus as a Data Source, letting you write PromQL queries and turn the results into line charts, dashboards, and alerting rules.

As a concrete example, say you want to know the P95 latency of /api/v1/ticket (95% of requests come in under this latency). In Grafana, you’d write a PromQL query like this:

histogram_quantile(0.95, sum by (le) (rate(
  http_server_requests_seconds_bucket{uri="/api/v1/ticket"}[5m])))

Breaking it down:

• http_server_requests_seconds_bucket{uri="/api/v1/ticket"}: locks onto the histogram bucket data for this API
• rate(...[5m]): computes the per-second request rate over the last 5 minutes, smoothing out the numbers so a single burst of requests doesn’t cause a wild jump
• histogram_quantile(0.95, ...): derives the actual P95 latency in seconds from the bucket distribution

In the resulting chart, the X axis is time (HH:MM:SS, using your browser’s local timezone by default), and you need to pay special attention to the unit on the Y axis — because the metric name is http_server_requests_seconds_bucket, the seconds in the middle is the unit. So 0.15 on the Y axis means 0.15 seconds (150 milliseconds), not “150.” If reading seconds doesn’t feel intuitive, you can change the Unit setting in the Grafana panel to seconds (s), and it’ll automatically convert it to a readable ms display for you.

Generally speaking, here’s a rough benchmark for judging API latency by P95:

More important than the number itself is the trend. If you see P95 latency climbing continuously over a short period (rather than a single spike), it usually means some resource is being overwhelmed — maybe the connection pool is maxed out, a cache invalidation is sending all requests to the DB, or the JVM has slipped into frequent GC. At that point, it helps to pull up a few more lines for cross-reference:

# QPS over the same period, to rule out this simply being caused by rising traffic
sum(rate(http_server_requests_seconds_count{uri="/api/v1/ticket"}[5m]))

# HikariCP connection pool utilization, to see if connections are exhausted
hikaricp_connections_active / hikaricp_connections_max

# JVM GC pause time, to see if GC is causing the latency spike
rate(jvm_gc_pause_seconds_sum[5m])

If QPS hasn’t changed much but latency keeps climbing, the culprit is most likely the connection pool or cache — not simply too much traffic. This is exactly the real value of Metrics visualization: it lets you spot the warning signs before you’re getting paged at 2am.

More Common Metrics

There are quite a lot of these — here are a few common ones:

# QPS over the same period, to rule out this simply being caused by rising traffic
sum(rate(http_server_requests_seconds_count{uri="/api/v1/ticket"}[5m]))

# HikariCP connection pool utilization, to see if connections are exhausted
hikaricp_connections_active / hikaricp_connections_max

# JVM GC pause time, to see if GC is causing the latency spike
rate(jvm_gc_pause_seconds_sum[5m])

# CPU usage (of this JVM process itself)
process_cpu_usage * 100

# CPU usage (at the whole-machine level) — keep this distinct from the one above; it matters who's actually doing the eating
system_cpu_usage * 100

# JVM Heap Memory utilization, to see whether it's about to fill up and whether -Xmx needs adjusting
sum(jvm_memory_used_bytes{area="heap"}) / sum(jvm_memory_max_bytes{area="heap"}) * 100

# Memory usage broken down by GC generation (Eden, Old Gen, etc.), very useful for tracking down memory leaks
jvm_memory_used_bytes{area="heap"}

# Number of live JVM threads — a sudden spike usually means a misconfigured Thread Pool or a thread leak
jvm_threads_live_threads

# Remaining disk space ratio — too low and /actuator/health will report DOWN directly
disk_free_bytes / disk_total_bytes * 100

# HTTP 5xx error rate — look at this alongside latency to know whether the system is "slow" or "broken"
sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
  / sum(rate(http_server_requests_seconds_count[5m])) * 100

There are a huge number of these metrics, so my suggestion is: keep a cheat sheet, and pull it out when you need it — no need to memorize them all.

To sum it all up — Spring Actuator + Grafana + Prometheus, used together, is what lets us pinpoint exactly where a system’s bottleneck actually is. In the next installment, we’ll bring in real-world scenarios and walk through a few practical cases. That’s it for this article.