Bootstrapping Spark 3.0.0 on EMR cluster

Question

A few days back Spark 3.0.0 was launched. I would like to use some of these functionalities. The default version for Spark on an EMR cluster now is Spark 2.4.5. I specifically make use of PySpark. My question is: how can I install/bootstrap Spark 3.0.0 on an EMR cluster?

Thanks in advance.

You wait for EMR to support it, or you run spark outside of EMR. — jordanm, Jun 22 '20 at 16:11
Alright that's unfortunate. I will wait then. Thanks for the comments! — thijsvdp, Jun 22 '20 at 16:20
This is not just installing spark. There will be lot of compatibility issues. Setting configuration will be also a tough job. — Snigdhajyoti, Jun 22 '20 at 17:05

score 1 · Answer 1 · answered Jun 22 '20 at 19:18

1

You'll need to wait for Spark 3 to be available on EMR. Judging by past releases you're probably looking at 1 - 3 months

answered Jun 22 '20 at 19:18

datamonk3y

124
1
4

Bootstrapping Spark 3.0.0 on EMR cluster

1 Answers1