Early Initialization of objects on Worker nodes in Spark Cluster

Question

I am using Drools with Spark in a stand alone cluster. I want to load the knowledge session on all the worker nodes at the startup i.e. before the map reduce task. I've tried passing the Statefull session from driver to slave nodes but its not working. Due to this my first job takes around 900ms just to add rules to Knowledge Builder.

Welcome to SO. Please read how to create a [mcve]. – zero323 Aug 01 '16 at 11:10 — zero323, Aug 01 '16 at 11:10

score 0 · Answer 1 · edited Jan 02 '17 at 10:09

Create Final variable of KnowledgeSession on driver,

final StatelessKnowledgeSession ksession = kbase.newStatelessKnowledgeSession();

And you can pass that ksession object to all the worker nodes,

 lines.foreachRDD(new Function<JavaRDD<String>, Void>() {
    public Void call(JavaRDD<String> rdd) throws Exception {
        List<String> facts = rdd.collect();
        //Apply rules on facts here
        ksession.execute(facts);
        return null;
    }
 });

Early Initialization of objects on Worker nodes in Spark Cluster

1 Answers1