0

I looked for the spark dependencies on mvnrepository.com, there are many versions of the spark core, in Central and Cloudera, Central has version 2.4.0 and Cloudera version 2.4.0.Clouder1 or 2.4.0.Clouder2

For example, in spark core dependencies, RoaringBitmap is version 0.5.11 in Central and 0.7.45 in Cloudera

If my pom dependencies are using the Central version I need to run it on a version of spark 2.4.0.cloudera. What can I need to do?

Vüsal
  • 2,580
  • 1
  • 12
  • 31
BlazerHe
  • 1
  • 2
    The difference is that the `Cloudera2` is another build of Spark done by **Cloudera**, in general they should be the _"same"_, but they do have small differences. If you are only going to deploy your app on a cloudera cluster, I would compile to the most specific version that you can. – Luis Miguel Mejía Suárez Sep 09 '19 at 11:50
  • @LuisMiguelMejíaSuárez If my app dependencies on RoaringBitmap [0.7.45], but the production environment's spark is 2.4.0.cloudera1 and RoaringBitmap is [0.5.11], how do I compile it? – BlazerHe Sep 10 '19 at 15:48
  • if **Spark** does not expose any class for that library. You can _shade_ your own version inside your path. That is a complex task, which I can not help with. I would suggest reading more about that, and if you have further problems, ask a new _(more specific)_ question. - Other option would be to use the same version that your Spark uses, which may be easier from the jar packaging, but would require code changes. _(note that if Spark exposes those classes, you can only go with this alternative, as shading will not work)_. – Luis Miguel Mejía Suárez Sep 11 '19 at 01:49

0 Answers0