error: value show is not a member of Unit CaseFileDFTemp.show()

Question

I ran below code in databricks scala notebook but I am getting error.

LIBRARY ADDED : azure-cosmosdb-spark_2.4.0_2.11-1.3.4-uber CODE :

import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}

import spark.implicits._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.Column
import org.apache.spark.sql.types.{StructType, StructField, StringType, IntegerType,LongType,FloatType,DoubleType, TimestampType}
import org.apache.spark.sql.cassandra._

//datastax Spark connector
import com.datastax.spark.connector._
import com.datastax.spark.connector.cql.CassandraConnector
import com.datastax.driver.core.{ConsistencyLevel, DataType}
import com.datastax.spark.connector.writer.WriteConf

//Azure Cosmos DB library for multiple retry
import com.microsoft.azure.cosmosdb.cassandra

import sqlContext.implicits._
spark.conf.set("x","x")
spark.conf.set("x","x")
spark.conf.set("x","x")
spark.conf.set("x","x")

val CaseFileDFTemp = sqlContext
  .read
  .format("org.apache.spark.sql.cassandra")
  .options(Map( "table" -> "case_files", "keyspace" -> "shared"))
.load().show()

CaseFileDFTemp.show()

ERROR:

error: value show is not a member of Unit CaseFileDFTemp.show()

score 1 · Answer 1 · edited Jan 10 '20 at 10:37

1

Can you please try creating the SQL context and try the show function.

import sqlContext.implicits._
val sqlContext= new org.apache.spark.sql.SQLContext(sc)

Please let me know if it helps.

edited Jan 10 '20 at 10:37

Rafaqat Ali

676
7
26

answered Jan 10 '20 at 07:32

Sukanya Arumugam

171
4

score 0 · Answer 2 · answered Jan 10 '20 at 13:38

If you write

val CaseFileDFTemp = sqlContext
  .read
  .format("org.apache.spark.sql.cassandra")
  .options(Map( "table" -> "case_files", "keyspace" -> "shared"))
.load().show()

Then CaseFileDFTemp will have type Unit, because the show() will "consume" your dataframe. So remove show(), then it will work

error: value show is not a member of Unit CaseFileDFTemp.show()

2 Answers2