28

I have tested hadoop and mapreduce with cloudera and I found it pretty cool, I thought I was the most recent and relevant BigData solution. But few days ago, I found this : https://spark.incubator.apache.org/

A "Lightning fast cluster computing system", able to work on the top of a Hadoop cluster, and apparently able to crush mapreduce. I saw that it worked more in RAM than mapreduce. I think that mapreduce is still relevant when you have to do cluster computing to overcome I/O problems you can have on a single machine. But since Spark can do the jobs that mapreduce do, and may be way more efficient on several operations, isn't it the end of MapReduce ? Or is there something more that MapReduce can do, or can MapReduce be more efficient than Spark in a certain context ?

user4157124
  • 2,809
  • 13
  • 27
  • 42
Nosk
  • 753
  • 2
  • 6
  • 24

2 Answers2

39

Depends what you want to do.

MapReduce's greatest strength is processing lots of large text files. Hadoop's implementation is built around string processing, and it's very I/O heavy.

The problem with MapReduce is that people see the easy parallelism hammer and everything starts to look like a nail. Unfortunately Hadoop's performance for anything other than processing large text files is terrible. If you write a decent parallel code you can often have it finish before Hadoop even spawns its first VM. I've seen differences of 100x in my own codes.

Spark eliminates a lot of Hadoop's overheads, such as the reliance on I/O for EVERYTHING. Instead it keeps everything in-memory. Great if you have enough memory, not so great if you don't.

Remember that Spark is an extension of Hadoop, not a replacement. If you use Hadoop to process logs, Spark probably won't help. If you have more complex, maybe tightly-coupled problems then Spark would help a lot. Also, you may like Spark's Scala interface for on-line computations.

Adam
  • 16,808
  • 7
  • 52
  • 98
  • great answer except for the the statement "Spark is an extension of Hadoop". While the most popular practice is using Spark on top of Hadoop Yarn, spark could run on MESOS which is not in Hadoop, or it could even run standalone. – pedram bashiri Nov 13 '19 at 19:19
27

MapReduce is batch oriented in nature. So, any frameworks on top of MR implementations like Hive and Pig are also batch oriented in nature. For iterative processing as in the case of Machine Learning and interactive analysis, Hadoop/MR doesn't meet the requirement. Here is a nice article from Cloudera on Why Spark which summarizes it very nicely.

It's not an end of MR. As of this writing Hadoop is much mature when compared to Spark and a lot of vendors support it. It will change over time. Cloudera has started including Spark in CDH and over time more and more vendors would be including it in their Big Data distribution and providing commercial support for it. We would see MR and Spark in parallel for foreseeable future.

Also with Hadoop 2 (aka YARN), MR and other models (including Spark) can be run on a single cluster. So, Hadoop is not going anywhere.

Praveen Sripati
  • 32,799
  • 16
  • 80
  • 117
  • 1
    MR pattern won't go anywhere, but it's the platforms that might come and go. MR pattern can be implemented on Spark also. – Praveen Sripati Mar 05 '14 at 07:42
  • 1
    Found this: http://gigaom.com/2014/02/27/as-mapreduce-fades-apache-spark-is-now-a-top-level-project/# Looks like spark may become the next computation engine of hadoop after more updates :) – Nosk Mar 05 '14 at 16:53
  • 3
    @PraveenSripati: "Batch oriented" and "iterative" are not mutually exclusive. You can run an iterative algorithm as a batch job. – stackoverflowuser2010 Jan 25 '16 at 22:46