Printing values in map function on the console in yarn deployment mode

Question

I developed a streaming application, the application have a map function such as bellow:

probeFileLines.map(x => { println(x._2.toString().take(1)); x._2.toString()})

I need to print x._.2 on my console in the driver program...so I used take method but it doesn't show anything on the console..I am running my application on yarn -client mode..there are some threads addressing how to do this for a RDD but my problem is something different..

you are missing action function at the end. Spark transformations are lazy and do not do anything until action is called. Solutions provided below suggest making use of print or collet; both action function to make your map code to run. http://spark.apache.org/docs/latest/programming-guide.html#rdd-operations — Amit Kumar, Sep 15 '16 at 06:21

score 2 · Answer 1 · answered Oct 09 '18 at 04:17

2

You can refer to following approach:

val inputRDDV2 = sc.parallelize(List(1,2,3,4))
val inputRDDV2Map = inputRDDV2.map(num=>num*2).collect()
println("Applying map() to new RDD: ========================> ")
inputRDDV2Map.foreach(println)

answered Oct 09 '18 at 04:17

Ananta Chandra Das

1,865
2
14
18

score 0 · Accepted Answer · answered Sep 15 '16 at 02:10

0

Would you be able to use Spark Streaming's print() function?

val numberToPrint = 10;
probeFileLines
  .map(x => {
    x._2.toString()
  })
  .print(numberToPrint);

answered Sep 15 '16 at 02:10

Matt D

3,055
1
18
17

score 0 · Answer 3 · answered Sep 15 '16 at 03:20

0

You can use the collect method over RDD to get all elements of RDD as an array at driver program. Once you get array using the collect method, you can print element of array by iterating over it.

answered Sep 15 '16 at 03:20

Hokam

924
7
19

Printing values in map function on the console in yarn deployment mode

3 Answers3