1

Would love some help on this issue that I am having which ive been stuck on for quite some time now. I have a class Test(id: Int, name: String, type: SaleType). Where SaleType can be an enum with either SALE or PURCHASE. I want to collect this dataframe to a List(Seq(Test))

I have a dataframe like this.

val ds = Seq(
        (0, "p"),
        (1, "s")
      ).toDF("id", "name")

If I try to add a column SaleType

.withColumn("SaleType",when(col("name") === "p", SaleType.PURCHASE)
.otherwise(SaleType.SALE)

This does not work because I the column type cannot be SaleType.

When I try to create a dataframe with this

   val ds = Seq(
        (0, "p"),
        (1, "s")
      ).toDF("id", "name")
       .withColumn("SaleType",when(col("name") === "p", "PURCHASE")
       .otherwise("SALE").as[Test].collect().toList()

I get an error since I cannot create these classes Test because my 3rd column is a string.

Is there a function that can make sure I can still create a list of Test objects?

I thought I could not include the type column, collect the DF, add the SaleType and then cast as [Type], but unsure how to do that.

Thanks!

Henri Thorpe
  • 25
  • 4
  • when you instantiate a Test object, why did you provide 2 arguments only? The class constructor has 3 arguments. – mck May 27 '21 at 14:12
  • Apologies I made a mistake there. When I have a Test object I cannot create a DF from it. – Henri Thorpe May 27 '21 at 14:35
  • You didn't share your `SaleType` definition, how is it defined? – Gaël J May 27 '21 at 16:31
  • One option would be to create an encoder for `SaleType` as described [here](https://stackoverflow.com/a/39661890/2129801) – werner May 27 '21 at 17:11

0 Answers0