Slow first call after restarting Spring Boot application

Question

We have a React - Spring Boot application using Postgres, Hibernate, Jackson and Spring Data Rest.

After each application restart, the first call to a post route hitting one of our back-end servers is long (more than 4 seconds). Every subsequent call to the same route hitting each server is under 100 ms.

Our goal is to guarantee that none of our users is ever impacted by these slow calls after each redeployment.

We are considering triggering the calls automatically after each deployment so that the application "warms up" and our clients do not have the long calls.

Since there are several back-end servers behind a load balancer, we would like to trigger these calls directly from the back-end and not from a client.

We would like understand better what can be happening so that we to this more efficiently: could it be Hibernate? Spring lazy-loading beans? Jackson?

Hibernate L2 cache is not activated.

We expect to have the same fast response time each time (< 100 ms) instead of the initial 4s calls.

Just hit the endpoints to warm it up. You're only talking about adding an extra few seconds to a deployment/restart - big deal. IMHO it's not worth solving the underlying reasons. — Bohemian, Aug 01 '19 at 15:54
@Bohemian I tend to agree with you, is there are best practice to trigger the calls within the application at startup? (OP is in my team) — Rod Daunoravicius, Aug 01 '19 at 16:07
@rod I'm not aware of any best practice approach, but standard software engineering best practice says put it somewhere outside the app. I would think the logical place would be the start up shell script. You might want to put the brakes on making your server visible to the load balancer to prevent it from taking transactions until it's warmed up. I would work out the average time/activity needed, then use double that. — Bohemian, Aug 01 '19 at 16:26

g3org3 · Accepted Answer · 2020-02-03T17:13:33.687

Quick update we followed @willermo's answer plus a couple of tips from another forum have led us in the right direction to fix the problem.

We logged the class loading using the -verbose:class flag, which made it clear that the problem were classes being lazy loaded at the moment of the first call.

To pre-load these classes, we used an ApplicationRunner to trigger the calls on application startup, as suggested by @willermo; this allowed us to deterministically warm up all the servers behind the load balancer with a single call for each.

There were a couple extra obstacles that were easy to fix:

Adding the ApplicationRunner broke all our tests, so we had to exclude it from the test profile.
We did not want to persist the effects on the DB of these "fake" calls, so we wrapped them in a transaction that we rollback in the end.

This was our final solution:

@Component
@Profile("!test")
public class AppStartupRunner implements ApplicationRunner {

  // [Constructor with injected dependencies]

  @Transactional
  @Override
  public void run(ApplicationArguments args) throws Exception {
    // [Make the calls]
    TransactionAspectSupport.currentTransactionStatus().setRollbackOnly();    
  }
}

I would suggest to not use an `ApplicationRunner`, but instead create a `@Component` implementing `ApplicationListener`. The component's `onApplicationEvent` function gets called when your application is ready to serve requests. Now, you can use your logic to do your initialization calls. — Sebi, Oct 29 '21 at 07:32

score 2 · Answer 2 · answered Aug 01 '19 at 16:44

You can try to make the warm up call using an application runner.

public interface ApplicationRunner

Interface used to indicate that a bean should run when it is contained within a SpringApplication. Multiple ApplicationRunner beans can be defined within the same application context and can be ordered using the Ordered interface or @Order annotation.

e.g.

@Component
public class AppStartupRunner implements ApplicationRunner {


  @Override
  public void run(ApplicationArguments args) throws Exception {
    System.out.println("Running");
    //Make the first call here 
  }
}

Application Runner

Thanks @willermo, this as sent us in the right direction for our problem. I'm posting our final solution as another answer. — Rod Daunoravicius, Aug 02 '19 at 15:18

score 2 · Answer 3 · answered Aug 01 '19 at 16:46

I would look at your JDBC pool. Is Spring waiting for that first request before it populates the pool? Creating database connections is an expensive operation.

If you are using a Tomcat pool, look at the initial-size and min-idle properties. For HikariCP pool, look at minimum-idle.

You should be able to observe in a test environment if JDBC connections are opened to your database before any calls are made to the application.

The built-in Spring Boot Actuator health check endpoint also has options for checking the status of the database and may help bootstrap the pool.

Thanks @jlar310, these seem like good leads, but we found that the culprit was Spring class loading. — Rod Daunoravicius, Aug 02 '19 at 15:19

score 0 · Answer 4 · answered Jan 24 '22 at 20:53

0

In my case, the DB connection pool was taking time for initialization on first request.

I have added minpoolsize=10 in database string, now the first request is as quick as the subsequent requests.

The default minpoolsize is 0.

answered Jan 24 '22 at 20:53

Akshay Shinde

159
8

Slow first call after restarting Spring Boot application

4 Answers4