Why eclipse thinks df.as[CaseClass] as an error in Scala Spark program?

Question

I'm trying to convert a dataframe to dataset using the syntax

case class Schema(...)
val ds = df.as[Schema]

So my code looks like

case class Rule(rule_on: String, rule_operator: String, rule_value: Int, rule_name: String)
val rules_ds = rules_df
   .select("rule_on", "rule_operator", "rule_value", "rule_name")
   .as[Rule]

But eclipse is highlighting .as[Rule] as error. Screen shots of the same as below.

How to resolve this issue? I know its not a Scala issue, as it works on command line.

Environment (as in Eclipse):

Scala - 2.11.11
Spark - 2.4.0
JRE - 1.8

have you tried to define your case-class top-level, especially out of `main`? — Raphael Roth, Feb 26 '19 at 11:58
@RaphaelRoth, thanks that worked. I'm new to Scala, need to know a lot. — Ajax1986, Feb 26 '19 at 13:15

score 0 · Accepted Answer · answered Feb 28 '19 at 09:15

As suggested by Raphael Roth (in comments) I defined case class outside main method and it works like charm.

Also other solution (without using case class) is to typecast the dataframe to dataset as below

import org.apache.spark.sql._
val ds: Dataset[Row] = df

The above solution was taken from here

Why eclipse thinks df.as[CaseClass] as an error in Scala Spark program?

1 Answers1