14

We would like to implement Hadoop on our system to improve its performance.

The process works like this: Hadoop will gather data from MySQL database then process it. The output will then be exported back to MySQL database.

Is this a good implementation? Will this improve our system's overall performance? What are the requirements and has this been done before? A good tutorial would really help.

Thanks

Jonar
  • 143
  • 1
  • 1
  • 5

5 Answers5

5

Sqoop is a tool designed to import data from relational databases into Hadoop

https://github.com/cloudera/sqoop/wiki/

and a video about it http://www.cloudera.com/blog/2009/12/hadoop-world-sqoop-database-import-for-hadoop/

Raj
  • 22,346
  • 14
  • 99
  • 142
Joe Stein
  • 1,255
  • 2
  • 11
  • 14
2

As mentioned by Joe, Sqoop is a great tool of the Hadoop ecosystem to import and export data from and to SQL databases such as MySQl.

If you need more complex integration of MySQL including e.g. filtering or tranformation, then you should use an integration framework or integration suite for this problem. Take a look at my presentation "Big Data beyond Hadoop - How to integrate ALL your data" for more information about how to use open source integration frameworks and integration suites with Hadoop.

Raj
  • 22,346
  • 14
  • 99
  • 142
Kai Wähner
  • 5,248
  • 4
  • 35
  • 33
2

Hadoop is used for batch based jobs mostly on large sized semi structured data.. Batch in the sense even the shortest jobs is in the order of magnitudes of minutes. What kind of performance problem you are facing? Is it based on data transformations or reporting. Depending on that this architecture may help or make things worse.

Raj
  • 22,346
  • 14
  • 99
  • 142
Sai Venkat
  • 1,208
  • 9
  • 16
1

Altough it is not a regular hadoop usage. It migh make sense in following scenario:
a) If you have good way to partition your data into the inputs (like existing partitioning).
b) The processing of each partition is relatively heavy. I would give the number of at least 10 seconds of CPU time per partition.
If both conditions are met - you will be able to apply any desired amount of CPU power to make your data processing.
If your are doing simple scan or aggregation - I think your will not gain anything. On other hand - if your are going to run some CPU intensive algorithms on each partition - then indeed your gain can be significant.
I would also mention a separate case- if your processing require massive data sorting. I do not think that MySQL will be good in sorting billions of records. Hadoop will do it.

Amar
  • 11,930
  • 5
  • 50
  • 73
David Gruzman
  • 7,900
  • 1
  • 28
  • 30
  • It is a regular Hadoop usage. See Lamda architecture. Using MySQL as a speed layer is normative (though other tools are also used). SQOOP wouldn't exist if this wasn't normative. – nick Feb 21 '14 at 22:01
  • Usually hadoop makes sense when we add massive scalability. Running dozens of mappers agains one MySQL will not give much gain. Usual usage is to have hadoop to preprocess and aggregate raw data, and then load into RDBMS... – David Gruzman Feb 23 '14 at 19:46
0

I agree with Sai. I'm using Hadoop with MySql only when needed. I export the table into CSV and upload it to HDFS to process data more quickly. If you want to persist your processed data, you will have to write a single-reducer job that will do some kind of batchinserts to improve the performance of insertion.
BUT that really depends on what kind of things you want to do.

Thomas Jungblut
  • 20,854
  • 6
  • 68
  • 91
  • 1
    I believe exporting it to csv file and then loading it into mysql will be faster than direct batch insertion of output into db from hadoop. Latter would be a jdbc call which is slow compared fileload. – Harsha Hulageri Feb 24 '11 at 07:59