STORE output to a single CSV?

Question

Currently, when I STORE into HDFS, it creates many part files.

Is there any way to store out to a single CSV file?

score 17 · Accepted Answer · answered Mar 29 '12 at 10:24

You can do this in a few ways:

To set the number of reducers for all Pig opeations, you can use the default_parallel property - but this means every single step will use a single reducer, decreasing throughput:

set default_parallel 1;
Prior to calling STORE, if one of the operations execute is (COGROUP, CROSS, DISTINCT, GROUP, JOIN (inner), JOIN (outer), and ORDER BY), then you can use the PARALLEL 1 keyword to denote the use of a single reducer to complete that command:

GROUP a BY grp PARALLEL 1;

See Pig Cookbook - Parallel Features for more information

I don't think this is ideal since you might get out of memory error with too few reducers on large output data. — FreeTymeKiyan, Sep 27 '17 at 18:32

score 15 · Answer 2 · answered Dec 25 '13 at 14:58

You can also use Hadoop's getmerge command to merge all those part-* files. This is only possible if you run your Pig scripts from the Pig shell (and not from Java).

This as an advantage over the proposed solution: as you can still use several reducers to process your data, so your job may run faster, especially if each reducer output few data.

grunt> fs -getmerge  <Pig output file> <local file>

STORE output to a single CSV?

2 Answers2

Linked