0

My requirement is to read csv file from src-resourses folder and convert them to DF .My code is as below. Genre and month are case class am using this to give structure to my list so that I can convert them to DF .In the below code in the place of _ what should I use so that I can get List[Genre] or List[Month] based on the fileName value which am going to pass.

trait IndexReader[T] {
  def read(filePath: String, sep: String,fileName:String): List[T] = {
    val stream: InputStream = getClass.getResourceAsStream("/" + filePath)
    val lines: Iterator[String] = scala.io.Source.fromInputStream(stream).getLines
    lines.map(line => {normalize(line, sep,fileName)}).toList
  }
  def normalize(line: String, sep: String,fileName:String): T
}

class BrandReader extends IndexReader[_] {
  override def normalize(line: String, sep: String,fileName:String):_ = {
    val lineSplit: Seq[String] = line.split(sep).toList
    val file = fileName match {
      case "indexGenre" => Genre(lineSplit(0), lineSplit(1))
      case "indexMonth" => Month(lineSplit(0), lineSplit(1))
    }
    file
  }
}
minnu
  • 57
  • 1
  • 8
  • 3
    Scala 3 will have union types, so you could use `Genre|Month` but, until then, use the trait or abstract class that is common to both. – jwvh Aug 18 '20 at 16:56

2 Answers2

1

Pattern matching can't return different types from different branches.

If you can define a trait extended by both case classes you can try

sealed trait GenreOrMonth
case class Genre(s: String, s1: String) extends GenreOrMonth
case class Month(s: String, s1: String) extends GenreOrMonth

class BrandReader extends IndexReader[GenreOrMonth] {
  override def normalize(line: String, sep: String, fileName: String): GenreOrMonth = ...
}

Otherwise return type is just Product

class BrandReader extends IndexReader[Product] {
  override def normalize(line: String, sep: String, fileName: String): Product = ...
}

or Any

class BrandReader extends IndexReader[Any] {
  override def normalize(line: String, sep: String, fileName: String): Any = ...
}

One more option is to use Either

case class Genre(s: String, s1: String)
case class Month(s: String, s1: String)

class BrandReader extends IndexReader[Either[Genre, Month]] {
  override def normalize(line: String, sep: String, fileName:String): Either[Genre, Month] = {
    val lineSplit: Seq[String] = line.split(sep).toList
    val file = fileName match {
      case "indexGenre" => Left(Genre(lineSplit(0), lineSplit(1)))
      case "indexMonth" => Right(Month(lineSplit(0), lineSplit(1)))
    }
    file
  }
}

How to define "type disjunction" (union types)?

Dmytro Mitin
  • 48,194
  • 3
  • 28
  • 66
  • Hi Dmytro,I tried the first solution sealed trait.when am calling the read method and converting the List[IndexFiles] to DF it is changing to nothing. val linesF: Map[String, Nothing] = file_path.map(x=>x._1 -> new BrandReader().read(file_path(x._1), file_sep,x._1).toDF()) – minnu Aug 19 '20 at 02:52
  • @minnu Can't reproduce. Your code snippet with `val linesF...` is not self-contained. If you need further assistance please provide self-contained piece of code. – Dmytro Mitin Aug 19 '20 at 11:00
  • when i call the read method as below val LinesF:List[GenreOrMonth] = new BrandReader().read("generafile.csv", ";","indexGenre") from this how can i convert to dataframe. when I add .toDF() it does not work – minnu Aug 20 '20 at 03:43
  • @minnu `.toDF()` works with `List[String]` and `new BrandReader().read(...)` is a `List[GenreOrMonth]`. With `List[Genre]` or `List[Month]` `.toDF()` didn't work either. How did you plan to make `.toDF()` work with `List[Genre]` or `List[Month]`? Do you mean `new BrandReader().read("generafile.csv", ";","indexGenre").map(_.toString).toDF()`? – Dmytro Mitin Aug 20 '20 at 07:57
1

Here's an idea that uses shapeless; I've simplified what you're doing so I can provide a runnable example. I show the code below, or click the link to see it in action.

https://scalafiddle.io/sf/BhyfPyu/1

import shapeless._

case class Genre(category:String, category2:String)
case class Month(month:String, year:String)

abstract class Reader {
  
  def read: (String,String) = ("String1","String2")
  
  val genTuple = Generic[(String,String)]

  def readT[T](implicit genT:Generic.Aux[T,String :: String :: HNil]): T= {
    val hList: (String :: String :: HNil) = genTuple.to(read)
    
    genT.from(hList)
  }
  
  def main(s:String) = {
    s match {
      case "GENRE" => readT[Genre]
      case "MONTH" => readT[Month]
      case _ => throw new IllegalArgumentException(s"$s not supported")
    }
  }
  
}


object ReaderImpl extends Reader

println(ReaderImpl.main("GENRE"))
println(ReaderImpl.main("MONTH"))
Nick
  • 645
  • 7
  • 22