Using Quartz with Mule in Clustered Environment

Question

I have a scenario where , I am trying to read data from Yelp API and want to put it into a ActiveMQ queue after certain intervals, So I am using quartz scheduler for the same.My quartz scheduler runs after every 10 minutes and pushes the data to queue, All is fine till here,

Now I want this to work in a clustered environment, where I will have 2 instances deployed and listening to same Yelp Endpoint , Now what is happening is, my quartz scheduler from 2 instances are executing at same instance and they extract same information from Yelp ,causing same messages to land up in ActiveMQ queue, that is DUPLICATES,(I want to use clustered environment for High availability purposes, i.e. if any node fails other node can takeover.)

So is there any configuration, in Mule which can promote one node as master and other as failover node.

Thanks for all the help!

Answer 1

This will trigger by the cron expression 0/10 * * * * ? (each 10th second) for one of all nodes running the same application and that connects to the same database (MySQL in this case). The Quartz setup is a bit messy. You need to configure the database etc, but I leave you studying the Quartz docs for that. You should look at version 1.8.x and not 2.x.

It's pretty much a budget alternative of clustering endpoints in Mule EE. It's useful when you do not want to cluster your EE nodes or need to run CE nodes.

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:quartz="http://www.mulesoft.org/schema/mule/quartz" xmlns:file="http://www.mulesoft.org/schema/mule/file" xmlns:tracking="http://www.mulesoft.org/schema/mule/ee/tracking" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
    xmlns:spring="http://www.springframework.org/schema/beans" version="EE-3.6.2"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd
http://www.mulesoft.org/schema/mule/ee/tracking http://www.mulesoft.org/schema/mule/ee/tracking/current/mule-tracking-ee.xsd
http://www.mulesoft.org/schema/mule/quartz http://www.mulesoft.org/schema/mule/quartz/current/mule-quartz.xsd">

    <quartz:connector name="quartzConnector" validateConnections="true" doc:name="Quartz">
        <quartz:factory-property key="org.quartz.scheduler.instanceName" value="QuartzScheduler" />
        <quartz:factory-property key="org.quartz.scheduler.instanceId" value="AUTO" />
        <quartz:factory-property key="org.quartz.jobStore.isClustered" value="true" />
        <quartz:factory-property key="org.quartz.scheduler.jobFactory.class" value="org.quartz.simpl.SimpleJobFactory" />
        <quartz:factory-property key="org.quartz.threadPool.class" value="org.quartz.simpl.SimpleThreadPool" />
        <quartz:factory-property key="org.quartz.threadPool.threadCount" value="3" />
        <quartz:factory-property key="org.quartz.scheduler.rmi.proxy" value="false" />
        <quartz:factory-property key="org.quartz.scheduler.rmi.export" value="false" />

        <quartz:factory-property key="org.quartz.jobStore.class" value="org.quartz.impl.jdbcjobstore.JobStoreTX" />
        <quartz:factory-property key="org.quartz.jobStore.driverDelegateClass" value="org.quartz.impl.jdbcjobstore.StdJDBCDelegate" />
        <quartz:factory-property key="org.quartz.jobStore.dataSource" value="quartzDataSource" />
        <quartz:factory-property key="org.quartz.jobStore.tablePrefix" value="QRTZ_" />

        <quartz:factory-property key="org.quartz.dataSource.quartzDataSource.driver"  value="com.mysql.jdbc.Driver" />

        <quartz:factory-property key="org.quartz.dataSource.quartzDataSource.URL" value="jdbc:mysql://localhost:3306/qrtz" />
        <quartz:factory-property key="org.quartz.dataSource.quartzDataSource.user" value="root" />
        <quartz:factory-property key="org.quartz.dataSource.quartzDataSource.password"  value="" />
        <quartz:factory-property key="org.quartz.dataSource.quartzDataSource.maxConnections"  value="8" /> 

    </quartz:connector>

    <flow name="cFlow1">
        <quartz:inbound-endpoint jobName="job1" cronExpression="0/10 * * * * ?" repeatInterval="0" connector-ref="quartzConnector" responseTimeout="10000" doc:name="Quartz">
            <quartz:event-generator-job>
                <quartz:payload>Job Trigger</quartz:payload>
            </quartz:event-generator-job>
        </quartz:inbound-endpoint>
        <logger level="INFO" message="Got message" doc:name="Logger"/>
    </flow>        
</mule>

Answer 2

We use 3.5.2-Enterprise edition but not sure if there is restriction on the community edition as such. Can you try the following way and see if that works:

<!-- Quart Connector with one thread to ensure that we don't have duplicate processing at any point of time -->
<quartz:connector name="QuartzConnector" validateConnections="true">
    <receiver-threading-profile maxThreadsActive="1" />
</quartz:connector> 

Then refer this in your flow wherever you are planning to trigger this action.

<flow name="test">
    <quartz:inbound-endpoint jobName="myQuartzJob" cronExpression="${my.job.cron.expression}" repeatInterval="${my.job.repeat.interval}" responseTimeout="${my.job.response.timeout}" connector-ref="QuartzConnector">
        <quartz:event-generator-job>
            <quartz:payload>blah</quartz:payload>
        </quartz:event-generator-job>
    </quartz:inbound-endpoint>
</flow>    

Hope that works.