0

Could someone please be so kind and tell me how to adapt the hdfs URIs in the following code so that they work against my local spark 'cluster'?

var lines = sparkContext.TextFile(@"hdfs://path/to/input.txt");  
// some more code
wordCounts.SaveAsTextFile(@"hdfs://path/to/wordcount.txt");
zero323
  • 322,348
  • 103
  • 959
  • 935
cs0815
  • 16,751
  • 45
  • 136
  • 299
  • Spark local mode does not involve a cluster. Are you running a cluster (Standalone, YARN or Mesos) in the machine where you are trying to run Mobius-based Spark application? – skaarthik Apr 16 '17 at 17:32
  • Sure it doesn't. That's why I used inverted commas. I simply want to develop a console application in visual studio against my local spark before I deploy it in the future. Yes, I am trying to run a Mobius-based Spark application – cs0815 Apr 16 '17 at 17:37

1 Answers1

0

You can just define path location config parameter will get setup on sparkcontext so no need to add hdfs just like below should be fine to run application in yarn mode

var lines = sparkContext.TextFile("/path/to/input.txt");  
// some more code
wordCounts.SaveAsTextFile("/path/to/wordcount.txt");

or you can to explicitly define hdfs location as below

val lines =  sparkContext.textFile("hdfs://namenode:port/path/to/input.txt")

you can also define number of partitions which is optional

var lines = sparkContext.TextFile("/path/to/input.txt",[number of partitions]);
Nitin
  • 3,533
  • 2
  • 26
  • 36
  • Thanks. (1) how do I parameterise the SparkContext to run against the local spark? (2) what would be the code if my file is here c:\war_and_peace.txt? – cs0815 Apr 16 '17 at 13:04