case classes in scala does not accept more than 22 parameters

Question

This is a challenge specific to SPARK-SQL and I'm unable to apply two highlighted answers

I'm writing complex data processing logic in SPARK-SQL.

Here is the process I follow ,

Define case class for a table with all attributes.
Register that as table.
Use SQLContext to query the same.

I'm encountering an issue as Scala allows only 22 parameters whereas my table has 50 columns. Only approach I could think of is to break dataset in such a way that it has 22 parameters and combine them later at the end. It does not look like a clean approach. Is there any better approach to this issue ?

Hi , This problem is slightly different as I'm applying this in SPARK-SQL. Spark does not build against SCALA 2.11 at the moment and creating a class (instead of case class) is not an option as I need to apply aggregate function and SPARK-SQL does not recognize this.. — user3279189, Aug 05 '14 at 10:25
http://mail-archives.apache.org/mod_mbox/spark-user/201407.mbox/%3CCAJ9c7gCcRUVvx0UX4ctq20m7=bPzxieP+JNjcsYsuyjcVpC8kQ@mail.gmail.com%3E — user3279189, Aug 05 '14 at 15:53

score 2 · Answer 1 · edited Aug 04 '14 at 15:53

2

Switch to Scala 2.11 and the case class field limit is gone. Release notes. Issue.

edited Aug 04 '14 at 15:53

Chris Martin

30,334
10
78
137

answered Aug 04 '14 at 15:48

Gangstead

4,152
22
35

1

Note that Tuples, anon. Function args. still have that limit, but yea, case classes eliminated it. – Dean Wampler Aug 04 '14 at 19:51
2

But Spark is not compatible with scala 2.11 version, how do you circumvent that ? – Soumitra Jan 20 '16 at 17:45

case classes in scala does not accept more than 22 parameters

1 Answers1