0

I know Shark has been subsumed by Spark SQL, a new module in Apache Spark. But my question is, can we use the existing Shark with new Spark versions ?

Devan M S
  • 692
  • 9
  • 23

1 Answers1

1

According to its release notes, Shark 0.9.1 only supports Spark 0.9.1.

According to a mailing list post, there is not an official version of Shark for Spark 1.0+. Shark depends on non-public interfaces of Spark that have likely changed between 0.9.1 and 1.1.0, so it may be a significant amount of work to port Shark to Spark 1.0+.

Josh Rosen
  • 13,511
  • 6
  • 58
  • 70
  • Thanks Josh. We tried shark 0.9.1 with Spark 1.1.0, as you mentioned it shows some conflicts with logging. May be spark updated the logging jar versions. – Devan M S Oct 15 '14 at 04:22