Merging small .bz2 compressed files on HDFS increases the total size

Asked Mar 19 '14 at 05:56

Active Jan 15 '15 at 18:56

Viewed 498 times

I have 300K small .bz2 files on HDFS. I am writing a pig job to merge all the files and produce 500 output .bz2 files. The total size of the small files were 300GB. After merging, the total size of the 500 merged files are around 500GB. This is the pig script I have used.

data = load 'inputFolder';   -- 300K files with total size 300GB
data = DISTINCT data PARALLEL 500; 
store data into 'outputFolder';  --500 files with total size 500GB

Can you explain me how this space is getting increased? Is there any alternate method on Pig to do the same?

Thanks, Tony

edited Jan 15 '15 at 18:56

James A Mohler

11,060
15
46
72

asked Mar 19 '14 at 05:56

user3436152

Are you sure the output is being written as bz2? The output filenames would need the bz2 extension, not just a directory name. – JamCon Mar 19 '14 at 19:35
Yes. Pig will write it as bz2. For eg: outputFolder2.bz2/part-r-00499.bz2 – user3436152 Mar 20 '14 at 03:47
Hey Guys,Do you have an input for me? – user3436152 Apr 01 '14 at 22:50

Merging small .bz2 compressed files on HDFS increases the total size

0 Answers0