Retry Quartz Scheduler start up on database connection failure

Question

We have a Java application that uses Quartz for scheduling jobs. The version of quartz that we use is: quartz-2.2.1

The quartz configuration uses JDBC job store.

If the database connection is down (due to intermittent network failure) at the time of calling start method on quartz scheduler object, it fails with the following exception:

2017-05-28 00:05:45 org.quartz.SchedulerConfigException: Failure occured during job recovery. [See nested exception: org.quartz.JobPersistenceException: Couldn't recover jobs: The connection is closed. [See nested exception: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.]]
2017-05-28 00:05:45     at org.quartz.impl.jdbcjobstore.JobStoreSupport.schedulerStarted(JobStoreSupport.java:692)
2017-05-28 00:05:45     at org.quartz.core.QuartzScheduler.start(QuartzScheduler.java:567)
2017-05-28 00:05:45     at org.quartz.impl.StdScheduler.start(StdScheduler.java:142)

In order to ensure that quartz scheduler is started successfully, we added retry in our code which makes a call to start method on quartz scheduler object after every 1 second. But when the database connection is up, the call to quartz scheduler start method is successful (it doesn't throw any exception) - but the associated triggers that reside in the database are not started and no job is triggered.

Any idea what could be the issue here? Any help shall be appreciated.

Here is the quartz configuration, please note that we have already enabled validation query to handle crappy connections (due to intermittent network failure)

#============================================================================
# Configure Main Scheduler Properties  
#============================================================================

org.quartz.scheduler.instanceName = TestScheduler
org.quartz.scheduler.instanceId = AUTO
org.quartz.scheduler.skipUpdateCheck=true

#============================================================================
# Configure ThreadPool  
#============================================================================

org.quartz.threadPool.class =     org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount =  50
org.quartz.threadPool.threadPriority = 5

#============================================================================
# Configure JobStore  
#============================================================================

org.quartz.jobStore.misfireThreshold = 60000
org.quartz.jobStore.maxMisfiresToHandleAtATime = 15

org.quartz.jobStore.class =     org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass =     org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.useProperties = false
org.quartz.jobStore.dataSource = aBPM
org.quartz.jobStore.tablePrefix = ABPM_
org.quartz.jobStore.isClustered = false
org.quartz.jobStore.useDBLocks = false
org.quartz.jobStore.acquireTriggersWithinLock = true

#============================================================================
# Configure Datasources  
#============================================================================
org.quartz.dataSource.aBPM.driver = org.hsqldb.jdbcDriver
org.quartz.dataSource.aBPM.URL = jdbc:hsqldb:file:embeddedDb/db/abpmquartz
org.quartz.dataSource.aBPM.user = sa
org.quartz.dataSource.aBPM.encryptPassword = yes
org.quartz.dataSource.aBPM.password = fMFVvEFk3gFmM9ewWQkTNg==
org.quartz.dataSource.aBPM.maxConnections = 55
org.quartz.dataSource.aBPM.validationQuery= SELECT 1

Hi Aman, it's been almost two months since you posted this question and I answered it. If you found my answer to be of any help, it'd be nice of you to mark it as accepted (the check under the voting arrows). Thanks! — walen, Aug 02 '17 at 12:21

score 3 · Answer 1 · answered Jun 08 '17 at 08:00

You might have an issue with job misfires.

TL;DR: use appropriate misfire instructions when building your triggers, or increase misfireThreshold.

If the Scheduler was down at the time the triggers were supposed to fire, that's a misfire. Once the Scheduler is up, Quartz checks for misfired jobs and looks for instructions about what to do with them. Run them immediately? Wait until their next scheduled fire time? To let it know what to do, you can either be explicit by using misfire instructions, or just default to Quartz's smart policy, which depends on the trigger type (e.g. CronTrigger has a different default misfire policy than SimpleTrigger).

Sadly, Quartz's cookbook is kind of lacking when it comes to explaining misfire instructions, and tells you to check the JavaDoc for every Trigger subclass if you want to know more.
So this one guy wrote a blog entry with all misfire instructions and default misfire policies explained for your convenience.

You didn't tell us what kind of triggers you are using, but you probably want to include misfire instructions like withMisfireHandlingInstructionFireNow(), which will run your job as soon as the Scheduler is up.

Another option is to set org.quartz.jobStore.misfireThreshold to a value higher than what the Scheduler takes to start.
Right now you have it set to 1 minute. That means that any job firing less than 1 minute later than its expected fire time, will not be considered a misfire and will be run just fine. However, for jobs that were more than 1 minute late, Quartz will check misfiring policy for them.

Say you know the Scheduler always takes less than 5 minutes to come online; you may then try to set org.quartz.jobStore.misfireThreshold = 300000, so when the jobs fire up on scheduler start, Quartz sees that they were late for less than 5 minutes and will just let them execute without checking the misfire policy first.

Retry Quartz Scheduler start up on database connection failure

1 Answers1