Questions tagged [apache-spark-encoders]
54 questions
0
votes
1 answer
Error: Unable to find encoder for type org.apache.spark.sql.Dataset[(String, Long)]
Following test for Dataset comparison is failing with the error:
Error:(55, 38) Unable to find encoder for type org.apache.spark.sql.Dataset[(String, Long)]. An implicit Encoder[org.apache.spark.sql.Dataset[(String, Long)]] is needed to store…
Aravind Yarram
- 78,777
- 46
- 231
- 327
0
votes
1 answer
Parsing Protobuf ByteString in Spark not working after creating Encoder
I'm trying to parse protobuf (protobuf3) data in spark 2.4 and I'm having some trouble with the ByteString type. I've created the case class using the ScalaPB library and loaded the jar into a spark shell. I've also tried creating a implicit encoder…
ukbaz
- 519
- 2
- 14
- 30
0
votes
1 answer
Spark Scala Dataset Type Hierarchy
Trying to enforce classes that extend W to have a method get that returns a Dataset of a subclass of a WR.
abstract class WR
case class TGWR(
a: String,
b: String
) extends WR
abstract class W {
def get[T <: WR]():…
devopslife
- 668
- 1
- 9
- 21
0
votes
0 answers
Why No Encoders For Pyspark
I am New to Pyspark world. I understand That Encoders are used by Spark SQl to serialize deserialize objects internally by Spark SQL. I know it is about converting Objects to spark sql internal storage types. I see lot of examples on internet about…
Harjeet Kumar
- 504
- 2
- 7
0
votes
1 answer
Why does creating a Dataset with LinearRegressionModel fail with "No Encoder found for org.apache.spark.ml.regression.LinearRegressionModel"?
I get a DataFrame contians Tuple(String, org.apache.spark.ml.regression.LinearRegressionModel):
val result = rows.map(row => {
val userid = row.getString(0)
val frame = filterByUserId(userid ,dataFrame)
(userid, lr.fit(frame,…
Pi Pi
- 861
- 3
- 13
- 22
0
votes
1 answer
Dataset encoder for scala collections
I get exception when trying to read dataset from s3. Company case class contains set with Employee case classe.
Exception in thread "main" java.lang.UnsupportedOperationException: No Encoder found for Set[com.model.company.common.Employee]
- field…
Przemek
- 208
- 2
- 8
0
votes
1 answer
How to set Encoder for Row, LabeledPointData in Spark?
How to set Encoders for LabeledPointData which is combination of Double, Vectors of Double. How to set Encoders for creating DataFrame?
public static Dataset convertRDDStringToLabeledPoint(Dataset data,String delimiter) {
…
Manikandan Balasubramanian
- 1,079
- 4
- 14
- 27
0
votes
1 answer
How to write an Encoder for a collection in Spark 2.1?
I have this:
import spark.implicits._
import org.apache.spark.sql.catalyst.encoders.RowEncoder
val mydata: Dataset[Row] = spark.read.format("csv").option("header", true).option("inferSchema", true).load("mydata.csv")
// CSV header:…
Randomize
- 8,651
- 18
- 78
- 133
-1
votes
1 answer
Value Type is binary after Spark Dataset mapGroups operation even return a String in the function
Environment:
Spark version: 2.3.0
Run Mode: Local
Java version: Java 8
The spark application trys to do the following
1) Convert input data into a Dataset[GenericRecord]
2) Group by the key propery of the GenericRecord
3) Using mapGroups after…
DeepNightTwo
- 4,809
- 8
- 46
- 60