1

Trying to create multiple dataframes in a single foreach, using spark, as below

I get values delivery and click out of row.getAs("type"), when I try to print them.

val check = eachrec.foreach(recrd => recrd.map(row => {
  row.getAs("type") match {
    case "delivery" => val delivery_data = delivery(row.get(0).toString,row.get(1).toString)
    case "click" => val click_data = delivery(row.get(0).toString,row.get(1).toString)
    case _ => "not sure if this impacts"
}})
)

but getting below error:

Error:(41, 14) type mismatch; found : String("delivery") required: Nothing case "delivery" => val delivery_data = delivery(row.get(0).toString,row.get(1).toString) ^

My plan is to create dataframe using todf() once I create these individual delivery objects referenced by delivery_data and click_data by:

delivery_data.toDF() and click_data.toDF().
  1. Please provide any clue regarding the error above (in match case).
  2. How can I create two df's using todf() in val check?
johnnyRose
  • 7,310
  • 17
  • 40
  • 61
satyajit vegesna
  • 99
  • 1
  • 1
  • 3

2 Answers2

0

val declarations make your first 2 cases return type to be unit, but in the third case you return a String

for instance, here the z type was inferred by the compiler, Unit:

def x = {
    val z: Unit = 3 match {
      case 2 => val a = 2
      case _ => val b = 3
    }
  }
pedrorijo91
  • 7,635
  • 9
  • 44
  • 82
-1

I think you need to cast this match clause to String. row.getAs("type").toString

user4925169
  • 1