2

I have a Spark Cluster running on Hadoop in YARN mode. I have configured Livy server to interact and submit client spark jobs to the spark cluster. I uploaded a data file along with the jar from the java program to Livy which gets uploaded in the .livySessions folder in HDFS. 

LivyClient client = new LivyClientBuilder(true).setURI(new URI("http://191.111.123.11:8998")).build();
client.uploadFile(new File("/C:/HAL/kmeans_data.txt")).get();
Object str = client.uploadJar(new File("/C:/Users/msingh/livy-spring-app-1.3.6.RELEASE.jar")).get();

How should I configure my program to read the data file from the current Livy session for execution.

mrsrinivas
  • 34,112
  • 13
  • 125
  • 125
msingh
  • 399
  • 2
  • 15

1 Answers1

1

You could use SparkFiles.get () which will fetch these files from within your spark job. Try and let me know.

Gururaj Srinivasa Rao
  • 11
  • 1