1

When trying to use Beam with spark 3.1.2 we are running into this issue :

InvalidClassException: scala.collection.mutable.WrappedArray

As explained here : https://www.mail-archive.com/issues@spark.apache.org/msg297820.html

It's an incompatibility issue :  spark 3.1.2 is compiled with scala 2.12.10 but this issue is fixed only for scala >= 2.12.14.

I see two solutions :

=> compiling spark & beam with a scala version >= 2.12.14

=> upgrading the spark version used by beam to 3.2.0, which can be compiled with scala 2.13

But both solutions could be long (need to understand beam source code), any idea ?

EDIT 21/01/2022:

My question is : do you think there is a simple workaround to this issue ?

I just found that a Jira ticket was already open about it : https://issues.apache.org/jira/browse/BEAM-12762

Thanks for your answer @Moritz but I don't see how to do it right now ; how can I package an other version of Sclala in the Beam Docker Image without compiling Beam ?

syronanm
  • 11
  • 2
  • Have you tried packaging the correct Scala version (e.g. 2.12.15) into your application jar? There's no need to compile Beam with the specific Spark and/or Scala version. The minor versions should be binary compatible. – Moritz Jan 12 '22 at 10:29
  • Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. – Community Jan 18 '22 at 18:06

0 Answers0