0

Not a programming question, but can someone enlighten me as to why Cloudera still only offer SPARK 1.6 on their Quickstart VM? What is the value of this when people download this for educational purposes. And they use Java 1.7 still.

tk421
  • 5,775
  • 6
  • 23
  • 34
thebluephantom
  • 16,458
  • 8
  • 40
  • 83
  • You can try asking on https://community.cloudera.com/ or file a JIRA https://issues.cloudera.org/secure/MyJiraHome.jspa to have them update their QuickstartVM. – tk421 Jul 31 '18 at 15:30
  • Well, thanks. But I did look there and found some poor answers. I thought that may be others have a better insight. In the bigger picture it is also very difficult to update that VM. So I install my own VM with SPARK 2 as that appears to be easier. Cheers – thebluephantom Jul 31 '18 at 15:40

2 Answers2

0

There is little reason to maintain 1.6 as the Cloudera Quickstart VM distribution.

Trying to install SPARK 2.2 gave all sorts of issues.

I am now using mapR v6 Sandbox that has this standardly but not IMPALA. My days with cloudera quickstart vm are behind me.

thebluephantom
  • 16,458
  • 8
  • 40
  • 83
0

Even I was surprised when I saw older version of spark in the VM. When I personally installed Cloudera 5.11 using parcels the default spark version was 1.6. Though it is present but it can be suppressed and Spark2.2 parcel can be used to enable Spark 2.2 on the same cloudera version. I have been using this combination seamlessly ever since I installed it. I use Spark to get data from Hive and persist the results back into Hive post processing in memory (using spark).

Prashant
  • 702
  • 6
  • 21
  • Well, on the quickstart VM I had all sorts of issues when rpmin the SPARK 2 .2 and when using the express option. So, I just switched to another distro. mapr has a number of things, except IMPALA, but I can live with that. I am expanding that. The doc on parcels etc. is fine, but we do not have parcel usage on quickstart vm from what I could understand that made it hard to follow. I think the issues is the way the quickstart is packaged. I used to be a DBA, knew once a lot on uNIX, re-acquiring these, so I am no so easily beaten... – thebluephantom Aug 13 '18 at 09:25
  • You are right, the VM is not updated properly with latest versions. – Prashant Aug 13 '18 at 09:47