0

I have read the Heron Documents about Implementing a Custom Scheduler. And I have known that I should implement some interfaces to implement a Custom Scheduler, such as ILauncher, IPacking, IScheduler and IUploader.

I have realized my CustomScheduler that implemented IScheduler interface, and I want to using LocalLauncher, LocalUploader and default Packing algorithm with my custom scheduler.

What's more, I modified the heron configuration file named scheduler.yaml that located at conf/local/ to use the custom scheduler. At the same time, I added the CustomScheduler.jar to heron-core/lib/scheduler/. However, there is something wrong as logs shows:

[2018-04-15 20:44:27 -0700] [STDERR] stderr: Exception in thread "main"   
[2018-04-15 20:44:27 -0700] [STDERR] stderr: com.twitter.heron.spi.scheduler.SchedulerException: Failed to instantiate scheduler using class 'com.zyt.heron.custom.scheduler.CustomScheduler'  
[2018-04-15 20:44:27 -0700] [STDERR] stderr:    at com.twitter.heron.scheduler.utils.LauncherUtils.getSchedulerInstance(LauncherUtils.java:120)  
[2018-04-15 20:44:27 -0700] [STDERR] stderr:    at com.twitter.heron.scheduler.SchedulerMain.runScheduler(SchedulerMain.java:382)  
[2018-04-15 20:44:27 -0700] [STDERR] stderr:    at com.twitter.heron.scheduler.SchedulerMain.main(SchedulerMain.java:218)

what should I do to fix this problem? Thanks for your help!

Yitian Zhang
  • 314
  • 1
  • 3
  • 18

3 Answers3

1

The direct reason for your Exception is your custom scheduler is not packaged with the heron release.

To make your custom scheduler work, you can follow the steps below:

  1. add your implementation into the directory heron/schedulers/src/java/com/twitter/heron/scheduler
  2. update the BUILD file at directory heron/schedulers/src/java
  3. add your custom scheduler building target at tools/rules/heron_client.bzl and tools/rules/heron_core.bzl

Here is an example of how local scheduler is packaged: https://github.com/apache/incubator-heron/search?utf8=%E2%9C%93&q=heron-local-scheduler&type=

Neng
  • 146
  • 2
1

The key is to provide your compiled .jar files to heron by copying your jar files into the .heron/lib/ folder.

I want to implement a custom scheduler and packing algorithm without recompiling Heron.

I have a project called "packing" that has the maven dependencies heron.api and heron.spi as requested.

My jar is called "packing-0.0.1-SNAPSHOT-jar-with-dendencies.jar built with maven. What works for me are the following steps summarized in a small shell-script. I run it from within my packing-project folder to the assembly-command works:

#!/bin/bash
mvn assembly:assembly
cp target/packing-0.0.1-SNAPSHOT-jar-with-dependencies.jar $HOME/.heron/lib/packing/
cp target/packing-0.0.1-SNAPSHOT-jar-with-dependencies.jar $HOME/.heron/lib/scheduler/
echo "updated packing plan and scheduler"

Then, simply add your custom packing or scheduler to your config:

Example from the packing.yaml: 

heron.class.packing.algorithm: com.hcep.packing.CustomPacking

with com.hcep.packing being my package structure in my packing-project and CustomPacking my packing class.

henriette_r
  • 56
  • 3
-1

You can develop custom scheduler for Heron - please follow any of the implementations for a scheduler in the code base.

Karthik
  • 132
  • 2