0

i am using Drools engine but only from the Implementation side. the frameWork is setup for me, and i can use only RULES side ( i hope i am able to explain myself).

that said - my problem is that i am trying to load about 1 milion row from Oracle DB into the WM, and i find that this task takes too long, here is the rule that i use to load the objects: (BTW - the request to load the milion records into the WM is mandatory since i need to use these DB objects as part of my rules along with other objects that are injected at runtime into the engine)

rule "Load_CMMObject" salience 10 no-loop

when

    not CMMObjectShouldBeRefreshed() over window:time( 1500m )

then
    log("Fetching Load_CMMObject");
    ExecutionContext ctx = getExecutionContext();


      String getObjectTypeSQL = "select AID , BC_OBJECT_ID , ALR_EQP_NAME , ALR_FROM_SITE from CMM_DB.COR_EQP_VW";
    PreparedStatement pStmt = null;

    try {

        pStmt = MTServiceAccessor.getDBAccessService().prepareNativeStatement(ctx, getObjectTypeSQL, false);
        ResultSet rs = pStmt.executeQuery();
        while (rs.next()) {

            String aid  = rs.getString(1);
            int objectID = rs.getInt(2);
            String  eqpName = rs.getString(3);
            String fromSite = rs.getString(4);


            CMMObject cmmObject = new CMMObject();

            cmmObject.setIp(aid);
            cmmObject.setObjectID(objectID);
            cmmObject.setEqpName(eqpName);
            cmmObject.setFromSite(fromSite);


            insert(cmmObject);
            //log("insert  Object --->  " + cmmObject.getIp());
                            }
        log("Finish Loading All cmm_db.BCMED_EQP_VW");
    } catch (Exception e) {
        log("Failed to Load ServiceName_TBL" + e);
    } finally {
        if (pStmt != null) {
            try {
                pStmt.close();
            } catch (Exception e) {
                log("Failed to close pstmt");
            }
        }
    }

    //log(" finished loading trails into the WM1");
    CMMObjectShouldBeRefreshed cMMObjectShouldBeRefreshed = new       CMMObjectShouldBeRefreshed();
    //log(" finished loading trails into the WM2");
    insert (cMMObjectShouldBeRefreshed);
    //log("finished loading trails into the WM3");

end

i am using server that allocates about 20Gb RAM for the Drools engine, and it has 8 1.5GHZ Quad Core proccessors.

the problem is that it take me to load 5000 raws about 1 minute --> so if i want to load the 1 milion records from the DB it will take me 200 minutes to complete the task and this is too much.

i will appreciate any help here,

thanks alot!

OrenT
  • 1
  • How long does it take to fetch the 1 million of records *without inserting them*? Is this fast enough? How many rules do you have? How long does it take to fire all rules for a single record? Multiply by one million: is this fast enough? Do you need all of the 1 million to be present in working memory *at the same time* for getting the correct results? – laune Jul 27 '15 at 18:39
  • Hi,what do you mean load record without inserting them? so what is the operation? – OrenT Jul 30 '15 at 06:25
  • Hi, I didnt finish my answer.. what kinf of operation besides insert, can i do to test this? further more - if you refer to fetch the same recored from oracle - it takes few minutes. as for the rules - even when i deactivate all rules and just perfrom the load operation - it takes approx the same time to load the objects into the WM. i do need all facts at the same time to get the needed result - i will explain the use case: – OrenT Jul 30 '15 at 06:35
  • The use case is:i have flow of facts comming into the Drools Engine. i need to calculate if these flow of facts have relations in the DB, at start there are about 150K of facts that i load immidetly - and for them i need to load from the DB all the needed relation which is about 1Milion rows. the information whether to link these facts - or generate more facts resides in the DB only - and i must load it to the WM to use it. the operation that i am doing for single row is simple: look for Fact X --> look for relation X-Y --> if there is no Fact Y present -then raise fact Y. – OrenT Jul 30 '15 at 06:36
  • To load a huge number of Ys just in order to learn whether there is a Y in relation to an X may be the wrong approach unless the number of Xs you need to process is about the same order of magnitude as the number of Ys. Your use case doesn't convince me that you really have to load 1M records up front. – laune Jul 31 '15 at 04:59

0 Answers0