1

I am trying to use pivot method in scala-spark 

val dfOutput =  df_input.groupBy("memberlogin").pivot("country_group2").count()

However, though there isn't any compliation error while creating a jar in eclipse, while execution in spark, its giving error as --
Exception in thread "main" java.lang.NoSuchMethodError: 

org.apache.spark.sql.GroupedData.pivot(Ljava/lang/String;)Lorg/apache/spark/sql/GroupedData;

I have imported the desired libraries, 

import org.apache.spark.sql.GroupedData
import org.apache.spark.sql.functions

also, 

import sqlContext.implicits._

as few other threads claimed.

Its not working, and unable to understand, can someone help or guide in direction?

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
rajendra patil
  • 31
  • 1
  • 6

2 Answers2

0

Pivoting feature was added in Spark 1.6 and it make possible creating pivot tables, with a DataFrame (with Scala, Java, or Python).

FaigB
  • 2,271
  • 1
  • 13
  • 22
0

Pivoting feature introduced in Spark 1.6 please check this documentation Reshaping Data with Pivot in Apache Spark . Check the API Pivot Api

Indrajit Swain
  • 1,505
  • 1
  • 15
  • 22