Quartz Job executed multiple times simultaneously by each cluster machine, rather than one time by one machine for the entire cluster

Question

Goal: * Have Job1 run once for a three-node cluster every 10 minutes, and Job2 run once for the same cluster every 5 minutes. Each job generates an email; so at 10:55am I should receive only one Job2 email from the cluster, and at 11:00am I should receive one Job1 email and one Job2 email from the cluster, at 11:05am I should receive only one Job2 email from the cluster, and so on...

Problem: * Job1 is being run multiple times every 10 minutes on each node in the cluster, and the same for Job2 (except every 5 minutes). This leads to many, many more than one or two emails.

Configuration: * Three-node linux cluster * Each machine NTP configured and time-sync'd * Oracle DB * Quartz v2.2.0 (cluster mode) * Jobs configured via CronTrigger * Each node has an instance of the same standalone Java application running on it, and the Java application instantiates an instance of the quartz scheduler in cluster-mode. * quartz.properties files are identical on each machine.

I have investigated all the obvious potential causes, but nothing explains it or presents a fix. I have even tried inserting an artificial 10-second sleep instruction in the job, to ensure that it doesn't finish in under a second. Please find relevant artifacts below (quartz.properties and log output). Any help would be greatly appreciated!

Artifact #1:

============================================================================
============================================================================
Q U A R T Z   ---   P R O P E R T I E S 
==================

    #============================================================================
    # Configure Main Scheduler Properties  
    #============================================================================

    org.quartz.scheduler.instanceName: MyQrtzScheduler
    org.quartz.scheduler.instanceId: AUTO

    org.quartz.scheduler.skipUpdateCheck: true

    #============================================================================
    # Configure ThreadPool  
    #============================================================================

    org.quartz.threadPool.class: org.quartz.simpl.SimpleThreadPool
    org.quartz.threadPool.threadCount: 1
    org.quartz.threadPool.threadPriority: 5

    #============================================================================
    # Configure JobStore  
    #============================================================================

    org.quartz.jobStore.misfireThreshold: 2592000000

    org.quartz.jobStore.class=org.quartz.impl.jdbcjobstore.JobStoreTX
    org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.oracle.OracleDelegate
    org.quartz.jobStore.useProperties=false
    org.quartz.jobStore.dataSource=myDS
    org.quartz.jobStore.tablePrefix=QRTZ_
    org.quartz.jobStore.isClustered=true
    org.quartz.jobStore.clusterCheckinInterval=60000

    #============================================================================
    # Other Example Delegates
    #============================================================================
    #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.DB2v6Delegate
    #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.DB2v7Delegate
    #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.DriverDelegate
    #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.HSQLDBDelegate
    #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.MSSQLDelegate
    #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.PointbaseDelegate
    #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
    #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.StdJDBCDelegate
    #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.WebLogicDelegate
    #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.oracle.OracleDelegate
    #org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.oracle.WebLogicOracleDelegate

    #============================================================================
    # Configure Datasources  
    #============================================================================

    org.quartz.dataSource.myDS.driver: oracle.jdbc.driver.OracleDriver
    org.quartz.dataSource.myDS.URL: jdbc:oracle:thin:@myServer:myPort:blah
    org.quartz.dataSource.myDS.user: myDBUser
    org.quartz.dataSource.myDS.password: myDBPassword
    org.quartz.dataSource.myDS.maxConnections: 2
    org.quartz.dataSource.myDS.validationQuery: select 0

    #============================================================================
    # Configure Plugins 
    #============================================================================

    org.quartz.plugin.shutdownHook.class: org.quartz.plugins.management.ShutdownHookPlugin
    org.quartz.plugin.shutdownHook.cleanShutdown: true
    org.quartz.plugin.triggerHistory.class=org.quartz.plugins.history.LoggingTriggerHistoryPlugin
    org.quartz.plugin.jobHistory.class=org.quartz.plugins.history.LoggingJobHistoryPlugin

Artifact #2:

============================================================================
============================================================================
L O G  ---  O U T P U T
==================
    2015-01-29 12:56:16,602 [main]  INFO com.mycompany.myapp.jobs.QuartzHelper - Initializing Quartz scheduler...
    2015-01-29 12:56:16,829 [main]  INFO org.quartz.impl.StdSchedulerFactory - Using default implementation for ThreadExecutor
    2015-01-29 12:56:16,855 [main]  INFO org.quartz.core.SchedulerSignalerImpl - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
    2015-01-29 12:56:16,855 [main]  INFO org.quartz.core.QuartzScheduler - Quartz Scheduler v.2.2.0 created.
    2015-01-29 12:56:16,857 [main]  INFO org.quartz.plugins.management.ShutdownHookPlugin - Registering Quartz shutdown hook.
    2015-01-29 12:56:16,859 [main]  INFO org.quartz.impl.jdbcjobstore.JobStoreTX - Using db table-based data access locking (synchronization).
    2015-01-29 12:56:16,864 [main]  INFO org.quartz.impl.jdbcjobstore.JobStoreTX - JobStoreTX initialized.
    2015-01-29 12:56:16,865 [main]  INFO org.quartz.core.QuartzScheduler - Scheduler meta-data: Quartz Scheduler (v2.2.0) 'MyQrtzScheduler' with instanceId 'node1_1422554176832'
      Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
      NOT STARTED.
      Currently in standby mode.
      Number of jobs executed: 0
      Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 1 threads.
      Using job-store 'org.quartz.impl.jdbcjobstore.JobStoreTX' - which supports persistence. and is clustered.

    2015-01-29 12:56:16,865 [main]  INFO org.quartz.impl.StdSchedulerFactory - Quartz scheduler 'MyQrtzScheduler' initialized from specified file: '/my/install/directory/quartz.properties'
    2015-01-29 12:56:16,866 [main]  INFO org.quartz.impl.StdSchedulerFactory - Quartz scheduler version: 2.2.0
    2015-01-29 12:56:16,866 [main]  INFO com.mycompany.myapp.jobs.QuartzHelper - Quartz scheduler initialized successfully.

    2015-01-29 12:59:53,450 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.core.QuartzSchedulerThread - batch acquisition of 1 triggers
    2015-01-29 13:00:00,007 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' is desired by: MyQrtzScheduler_QuartzSchedulerThread
    2015-01-29 13:00:00,008 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' is being obtained: MyQrtzScheduler_QuartzSchedulerThread
    2015-01-29 13:00:00,809 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' given to: MyQrtzScheduler_QuartzSchedulerThread
    2015-01-29 13:00:00,836 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' returned by: MyQrtzScheduler_QuartzSchedulerThread
    2015-01-29 13:00:00,839 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.simpl.PropertySettingJobFactory - Producing instance of Job 'node2_1422546730757.Job1', class=com.mycompany.myapp.job.Job1
    2015-01-29 13:00:00,851 [MyQrtzScheduler_Worker-1]  INFO org.quartz.plugins.history.LoggingTriggerHistoryPlugin - Trigger node2_1422546730757.Job1Trigger fired job node2_1422546730757.Job1 at:  13:00:00 01/29/2015
    2015-01-29 13:00:00,852 [MyQrtzScheduler_Worker-1]  INFO org.quartz.plugins.history.LoggingJobHistoryPlugin - Job node2_1422546730757.Job1 fired (by trigger node2_1422546730757.Job1Trigger) at:  13:00:00 01/29/2015
    2015-01-29 13:00:00,852 [MyQrtzScheduler_Worker-1] DEBUG org.quartz.core.JobRunShell - Calling execute on job node2_1422546730757.Job1
    2015-01-29 13:00:00,853 [MyQrtzScheduler_Worker-1]  INFO com.mycompany.myapp.job.Job1 - ***Executing Inbound File SLA Job...
    2015-01-29 13:00:02,054 [MyQrtzScheduler_Worker-1]  INFO com.mycompany.myapp.job.Job1 - ***Inbound File SLA Job: No SLA breaches found...
    2015-01-29 13:00:02,150 [MyQrtzScheduler_Worker-1]  INFO com.mycompany.myapp.job.Job1 - Job1 completed successfully in [1297ms]; sleeping [63703ms] to meet the required minimum runtime for quartz-jobs
    2015-01-29 13:00:24,881 [QuartzScheduler_MyQrtzScheduler-node1_1422554176832_ClusterManager] DEBUG org.quartz.impl.jdbcjobstore.JobStoreTX - ClusterManager: Check-in complete.
    2015-01-29 13:01:05,862 [MyQrtzScheduler_Worker-1]  INFO com.mycompany.myapp.job.Job1 - Job1 sleep-delay completed.
    2015-01-29 13:01:05,864 [MyQrtzScheduler_Worker-1]  INFO org.quartz.plugins.history.LoggingJobHistoryPlugin - Job node2_1422546730757.Job1 execution complete at  13:01:05 01/29/2015 and reports: SUCCESS
    2015-01-29 13:01:05,865 [MyQrtzScheduler_Worker-1]  INFO org.quartz.plugins.history.LoggingTriggerHistoryPlugin - Trigger node2_1422546730757.Job1Trigger completed firing job node2_1422546730757.Job1 at  13:01:05 01/29/2015 with resulting trigger instruction code: DO NOTHING
    2015-01-29 13:01:05,868 [MyQrtzScheduler_Worker-1] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' is desired by: MyQrtzScheduler_Worker-1
    2015-01-29 13:01:05,869 [MyQrtzScheduler_Worker-1] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' is being obtained: MyQrtzScheduler_Worker-1
    2015-01-29 13:01:05,872 [MyQrtzScheduler_Worker-1] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' given to: MyQrtzScheduler_Worker-1
    2015-01-29 13:01:05,880 [MyQrtzScheduler_Worker-1] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' returned by: MyQrtzScheduler_Worker-1
    2015-01-29 13:01:05,915 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.core.QuartzSchedulerThread - batch acquisition of 1 triggers
    2015-01-29 13:01:05,917 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' is desired by: MyQrtzScheduler_QuartzSchedulerThread
    2015-01-29 13:01:05,918 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' is being obtained: MyQrtzScheduler_QuartzSchedulerThread
    2015-01-29 13:01:05,921 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' given to: MyQrtzScheduler_QuartzSchedulerThread
    2015-01-29 13:01:05,954 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.impl.jdbcjobstore.StdRowLockSemaphore - Lock 'TRIGGER_ACCESS' returned by: MyQrtzScheduler_QuartzSchedulerThread
    2015-01-29 13:01:05,955 [MyQrtzScheduler_QuartzSchedulerThread] DEBUG org.quartz.simpl.PropertySettingJobFactory - Producing instance of Job 'node1_1422543657050.Job2', class=com.mycompany.myapp.jobs.Job2
    2015-01-29 13:01:05,961 [MyQrtzScheduler_Worker-1]  INFO org.quartz.plugins.history.LoggingTriggerHistoryPlugin - Trigger node1_1422543657050.Job2Trigger fired job node1_1422543657050.Job2 at:  13:01:05 01/29/2015
    2015-01-29 13:01:05,962 [MyQrtzScheduler_Worker-1]  INFO org.quartz.plugins.history.LoggingJobHistoryPlugin - Job node1_1422543657050.Job2 fired (by trigger node1_1422543657050.Job2Trigger) at:  13:01:05 01/29/2015
    2015-01-29 13:01:05,963 [MyQrtzScheduler_Worker-1] DEBUG org.quartz.core.JobRunShell - Calling execute on job node1_1422543657050.Job2
    2015-01-29 13:01:05,963 [MyQrtzScheduler_Worker-1]  WARN com.mycompany.myapp.jobs.Job2 - No outbound files found; Outbound File SLA Job cannot check for SLA breaches.
    2015-01-29 13:01:05,965 [MyQrtzScheduler_Worker-1]  INFO org.quartz.plugins.history.LoggingJobHistoryPlugin - Job node1_1422543657050.Job2 execution complete at  13:01:05 01/29/2015 and reports: null
    2015-01-29 13:01:05,966 [MyQrtzScheduler_Worker-1]  INFO org.quartz.plugins.history.LoggingTriggerHistoryPlugin - Trigger node1_1422543657050.Job2Trigger completed firing job node1_1422543657050.Job2 at  13:01:05 01/29/2015 with resulting trigger instruction code: DO NOTHING

Check [this question](http://stackoverflow.com/questions/11620979/quartz-scheduler-clustering-job-executing-twice) and also [this question](http://stackoverflow.com/questions/23665688/quartz-scheduler-trigger-some-jobs-on-every-cluster-node-and-some-only-once-per). — gknicker, Jan 29 '15 at 21:08
Thanks but unfortunately, those questions are tackling a different problem. — user3183901, Jan 29 '15 at 21:37

Alexander Suraphel · Answer 1 · 2017-05-20T17:06:21.223

The following answer was given by the OP.

The problem was that I was defining quartz jobs with identities that have a unique group id (the scheduler id) instead of a group id common to all hosts in the cluster. Since the scheduler id is unique to the host, each host in the cluster would look to see if that job already existed using the fully qualified job name groupId.jobName and surely it found it didn't, so it would create a new instance of Job1 and Job2 during startup. The quartz jobs/triggers are never expired or cleared without an explicit request in Java or manual sql statement in Oracle. So over time the instances would build up, and instead of quartz running a single instance of Job1 and Job2, it would run all the instances of each job that had been created over time (hence the multiple executions and multiple email alerts).

The solution is that I replace schedulerId with a static string such as "MyQuartzJobs" when defining a job's identity.

Basically, I changed the following line of Java code:

JobDetail job = 
newJob(Job1.class).withIdentity(JOB1_JOB_NAME,  uniqueSchedulerId)
.withDescription(JOB1_DESC + " created [" + new Date() + "]")
.storeDurably(false)
.requestRecovery(false)
.build();

to something like the following:

JobDetail job = 
newJob(Job1.class).withIdentity(JOB1_JOB_NAME,  "MyQuartzJobs")
.withDescription(JOB1_DESC + " created [" + new Date() + "]")
.storeDurably(false)
.requestRecovery(false)
.build();

Quartz Job executed multiple times simultaneously by each cluster machine, rather than one time by one machine for the entire cluster

1 Answers1