0

I have a little problem with sparkSubmit, when I send my job, I would like to recover the appId to store it somewhere. But I can not retrieve my id.

All sparkLauncher parameters are valid and the job runs well.

    try{
        LOGGER.debug("Start submit SparkJob");

        try {
            sparkAppHandle = sparkLauncher.startApplication();

            boolean write = false;
            long limitTime = waitingTime /100;

            LOGGER.debug("waiting time : "+waitingTime+"ms");

            int i=0;
            //log
            while( i <= limitTime ){
                Thread.sleep(100);
                i++;
                String id = sparkAppHandle.getAppId();
                if(StringUtils.isNotBlank(id)){
                    LOGGER.debug("i="+(i*100)+"ms,  SPARK id :" + id + ",  SPARK state :" + sparkAppHandle.getState().name());
                    write = true;
                }
            }
            if(!write){
                LOGGER.debug("SPARK id is null");
            }

        } catch (Exception e) {
            LOGGER.debug("ERROR in startApplication " + e);

        }
        LOGGER.debug("End submit SparkJob");

    } catch (Exception e) {
        LOGGER.error("Error in Spark Launcher:", e);
    }

Logs:

#09 Mar 2017 17:30:38,745  [DEBUG] (SparkServicesImpl.java:submitSparkJobsWithAppLaunch:143): Start submit SparkJob
#09 Mar 2017 17:30:38,774  [DEBUG] (SparkServicesImpl.java:submitSparkJobsWithAppLaunch:163): waiting time : 60000ms
#09 Mar 2017 17:31:38,942  [DEBUG] (SparkServicesImpl.java:submitSparkJobsWithAppLaunch:177): SPARK id null
#09 Mar 2017 17:31:38,942  [DEBUG] (SparkServicesImpl.java:submitSparkJobsWithAppLaunch:201): End submit SparkJob

My job runs normally, everything works with "sparkLauncher.launch ();" but I do not think I can recover the appId with this method.

Can someone help me ?

marchat
  • 153
  • 1
  • 9

1 Answers1

0

try SparkAppHandle.Listener you can get application id in infoChanged

user8117923
  • 1