Cannot update Heron topology using SchedulerStateManagerAdaptor

Question

When I attempted to create custom scheduler of Heron, and used SchedulerStateManagerApaptor object to update topology, this Exception happened:

Exception in thread java.util.ConcurrentModificationException: The update lock can not be obtained for topology AuroraMonitorSentenceWordCountTopology. Another actor is performing an update on it. Failing this request, try again once current update is complete
    at com.twitter.heron.scheduler.UpdateTopologyManager.updateTopology(UpdateTopologyManager.java:117)
    at zyt.custom.my.scheduler.aurora.AuroraHotEdgeSchedulerWithTxtLog.triggerSchedule(AuroraHotEdgeSchedulerWithTxtLog.java:323)
    at zyt.custom.my.scheduler.aurora.AuroraHotEdgeSchedulerWithTxtLog.access$400(AuroraHotEdgeSchedulerWithTxtLog.java:55)
    at zyt.custom.my.scheduler.aurora.AuroraHotEdgeSchedulerWithTxtLog$2.run(AuroraHotEdgeSchedulerWithTxtLog.java:258)
    at java.lang.Thread.run(Thread.java:748)

And the code of instantiating SchedulerStateManagerAdaptor object as follows:

Config config = Config.toClusterMode(this.config);
String stateMgrClass = Context.stateManagerClass(config); // get state manager instance
IStateManager stateMgr = null;
try {
    stateMgr = ReflectionUtils.newInstance(stateMgrClass);
} catch (ClassNotFoundException | InstantiationException | IllegalAccessException e) {
    e.printStackTrace();
}
stateMgr.initialize(config);
return stateManagerAdaptor = new SchedulerStateManagerAdaptor(stateMgr, 1000); // 5000: timeout

How to fix this exception? Thanks for your help!

score 0 · Answer 1 · answered Jul 03 '18 at 16:16

0

Yitan - Are you trying to create a new Aurora scheduler?

answered Jul 03 '18 at 16:16

Karthik

132
2

Yes, I am trying to create a Custom scheduler based on the Default Aurora Scheduler. And now, I want to be able to implement a feature that can be rescheduled after creating a job using the default aurora scheduler. But I have encountered the above problem. – Yitian Zhang Jul 03 '18 at 16:25

score 0 · Answer 2 · answered Jul 03 '18 at 18:17

0

Based on the exception message, there's one update operation going on when a second update operation is scheduled. The first on-going operation prevents the second one proceeding.

So could you try the update operation cleanly? And also please check if your implementation correctly releases locks once an update is done?

answered Jul 03 '18 at 18:17

Neng

146
2

This update operation is my first update process after default created job by AuroraScheduler. Before this update operation, I didn't execute any update operation. So I don't know who is taking the lock and how to release the lock correctly? – Yitian Zhang Jul 04 '18 at 03:27

Cannot update Heron topology using SchedulerStateManagerAdaptor

2 Answers2