To be used for Apache Spark 3.x
Tag is for all related to Apache Spark 3.0.0 and higher.
This tag is separate from apache-spark tag as this version has breaking changes.
Apache Spark is a unified analytics engine for large-scale data processing.
Questions tagged [spark3]
80 questions
0
votes
1 answer
How to port UDAF to Aggregator?
I have a DF looking like this:
time,channel,value
0,foo,5
0,bar,23
100,foo,42
...
I want a DF like this:
time,foo,bar
0,5,23
100,42,...
In Spark 2, I did it with a UDAF like this:
case class ColumnBuilderUDAF(channels: Seq[String]) extends…
rabejens
- 7,594
- 11
- 56
- 104
0
votes
2 answers
Pyspark SelectExp() not working for first() and last()
i have 2 statements which are to my knowledge exactly alike, but select() works fine, but selectExpr() generates following results.
+-----------------------+----------------------+
|first(StockCode, false)|last(StockCode,…
noobie-php
- 6,817
- 15
- 54
- 101
0
votes
1 answer
get_json_obj _fails for SelectExpr() but works for Select in Pyspark
i am facing a strange issue , i am trying to display values of my JSON object, it works fine with select() but it dont work with selectExp(), i get a weird error, following in my implementation,
from pyspark.sql import SparkSession
from…
noobie-php
- 6,817
- 15
- 54
- 101
0
votes
1 answer
Issue/Bug when loading and applying MultilayerPerceptronClassifier in Spark Version 3.0.0
IllegalArgumentException: MultilayerPerceptronClassifier_... parameter solver given invalid value auto
I believe I have discovered a bug when loading MultilayerPerceptronClassificationModel in spark 3.0.0, scala 2.1.2 which I have tested and can see…
st_60
- 1
- 2
-1
votes
1 answer
Spark3 Vector UDF vs Regular UDF
What is the difference between udf and vector udf in spark 3 as vectorized udf is new feature as per spark documentation
I know in In Spark 3, a user-defined function (UDF) is a function that you can define in a programming language such as Python…
Ram Ghadiyaram
- 28,239
- 13
- 95
- 121