0

This question has been asked here, but i don't think there is an answer.

My env: Spark v 2.2.1 and Scala 2.11.8

As shown in the original question, it seems spark-shell needs fully qualified notation 

import java.sql.Timestamp

case class Crime(
  caseNumber: String, date: Timestamp, 
  description: String, detail: String, 
  arrest: Boolean
)

//<console>:12: error: not found: type Timestamp
//         caseNumber: String, date: Timestamp,
//                                   ^

However, if the Timestamp is fully qualified, there is no issue

case class Crime(
  caseNumber: String, date: java.sql.Timestamp, 
  description: String, detail: String, 
  arrest: Boolean
)

// defined class Crime

And even for things like org.apache.spark.sql.Dataset or org.apache.spark.sql.functions.{lit, col}, importing doesn't work.

Any idea why? and it it possible to avoid the fully qualified notation?

PS: Databricks seems to not impose this constraint.

mrbrahman
  • 455
  • 5
  • 18
  • Does that mean that it works when you type `date: java.sql.Timestamp`? – ernest_k Mar 16 '18 at 17:35
  • Correct, I modified the question to clarify this. Thanks. – mrbrahman Mar 16 '18 at 17:41
  • Don't know why it's not working. But same thing is working in normal scala shell. Managed to make it work in **one CLI command**: `import java.sql.Timestamp;case class Crime(caseNumber: String, date: Timestamp, ...)`. Maybe you can try that – ernest_k Mar 16 '18 at 17:55
  • Spark has bug reports for this (e.g. [SPARK-17103](https://issues.apache.org/jira/browse/SPARK-17103)) but it never seems to have been thoroughly solved. The workaround suggested in the post you referenced (using `:paste`) seems to be the most reasonable one. – Tzach Zohar Mar 16 '18 at 18:06
  • This is a problem with the REPL since Scala 2.11. It works fine if you define your classes in a separate JAR. A workaround is to wrap your case classes in an object: `object Classes {case class...}` and then `import Classes._`. This way won't work for Datasets though. Related issue: [SPARK-18880](https://issues.apache.org/jira/browse/SPARK-18880?jql=text%20~%20%22shell%20case%20class%22) – Miguel Mar 16 '18 at 18:06
  • @TzachZohar, Miguel, thank you! Are you aware of any similar work around in Zepplin? – mrbrahman Mar 16 '18 at 18:24

0 Answers0