Spring Boot - running scheduled jobs as separate process

Question

I have a spring boot application which also have few scheduled jobs. I don't see any functional issue with implementation. one of the job runs every almost second for real time updates. There are other jobs also.

I suspect there is performance issue especially when long running API hits the controller.

// Heavy Job
@Scheduled(fixedRate = 10000)
public void processAlerts(){
}


@Scheduled(fixedDelayString = "${process.events.interval}")
public void triggerTaskReadiness() throws IOException {
    log.info("Trigger event processing job");
}
// Heavy Job to process data from different tables.
@Scheduled(fixedDelayString = "${app.status.interval}")
public void triggerUpdateAppHealth() throws IOException {
    log.info("Trigger application health");
}

Is it possible to have jobs as separate process. What are the best practices to have spring boot application with heavy jobs.

Answer 1

The question is way too general, IMO. It all depends on your resources and what exactly does the job do.

Spring boot provides a general purpose scheduling mechanism but doesn't make any assumptions about the job nature.

All-in-all, its true that when you run a heavy job, CPU, network, I/O and whatever resources are consumed (Again, depending on the actual code of your job).

If you run it externally basically another process will consume the same resources assuming its being run on the same server.

From the spring boot standpoint I can say the following:

1. It looks like the job deal with database. In this case Spring boot supports the integration with DataSources, connection pooling, transaction management, more High Level APIs like JPA or even spring data, you can also plug in frameworks like JOOQ. Bottom line, it makes the actual work with the database much easier.

You've stated Mongodb in the question tag - well, spring has also mongo db integration in spring data.

Bottom line if you're running the job in an external process you're kind of on your own (which doesn't mean it can't be done, it just means you lose all the goodies spring has upon its sleeves)

2. AppHealth - spring boot already provides an actuator feature that has an endpoint of db health, it also provides a way to create your own endpoints to check the health of any concrete resource (you implement it in code so you have a freedom to check however you want). Make sure you're using the right tool for the right job.

3. Regarding the controller API. If you're running with traditional spring mvc, tomcat has a thread pool to serve the API, so from the Threads management point of view the threads of job won't be competing with the threads of controller, however they'll likely share the same db connection so it can become a bottleneck.

4. Regarding the implementation of @Scheduled. By default there will be one thread to serve all the @Scheduled jobs, which might be insufficient.

You can alter this behavior by creating your own taskScheduler:

@Bean(destroyMethod = "shutdown")
public Executor taskScheduler() {
    return Executors.newScheduledThreadPool(10); // allocate 10 threads to run @Scheduled jobs
}

You might be interested to read this discussion

5. Spring @Scheduled always works "within the boundaries" of one spring managed application context. So that if you have decided to scale out your instances each and every instance will run the "scheduled" code and will execute heavy jobs. Its possible to use Quartz with which spring can be integrated do to clustered mode you can configure it to pick one node every time and execute the job, but since you're planning to run every second, I doubt quartz will work good enough.

6. A general observation: running a set of "heavy" jobs as you say doesn't really sounds well with "running every second". It just doesn't sound reasonable, since heavy jobs tend to last much longer than 1 second, so doing this will eventually occupy all the resources and you won't be able to run more jobs.