mrjob limit filesize for output files

Asked Jul 11 '16 at 22:15

Active Jul 11 '16 at 22:15

Viewed 108 times

Does anyone know how to limit the max size of s3 output files (part-r-00000, part-r-00001 ... etc) from mrjob?

I'm compressing the output if that makes any difference using the following in my .mrjob.conf file:

jobconf:
 mapred.output.compress: 'true'
 mapred.output.compression.codec: org.apache.hadoop.io.compress.GzipCodec

Thanks in advance, Conor

asked Jul 11 '16 at 22:15

Digan

Does this answer your question: http://stackoverflow.com/questions/15610116/how-to-limit-size-of-hadoop-sequence-file – Binary Nerd Jul 12 '16 at 07:26
Yes- thank you very much. :) – Digan Jul 12 '16 at 15:30

0 Answers0