3

I have created a folder to drop the result file from a Pig process using the Store command. It works the first time, but the second time it compains that the folder already exists. What is the best practice for this situiation? Documentation is sparse on this topic.

My next step will be to rename the folder to the original file name, to reduce the impact of this. Any thoughts?

mytwocents
  • 847
  • 17
  • 29
  • Possible duplicate of [How to force STORE (overwrite) to HDFS in Pig?](http://stackoverflow.com/questions/11110403/how-to-force-store-overwrite-to-hdfs-in-pig) – C8H10N4O2 Nov 23 '16 at 21:50

1 Answers1

8

You can execute fs commands from within Pig, and should be able to delete the directory by issuing a fs -rmr command before running the STORE command:

fs -rmr dir
STORE A into 'dir' using PigStorage();

The only subtly is the fs command doesn't expect quotes around the directory name, whereas the store command does expect quotes around the directory name.

Chris White
  • 29,949
  • 4
  • 71
  • 93
  • That just might work, to reduce possible conditions, I will name the folder the same as the file, so the last file of the same name to process will not fail on "folder exists" - thank you – mytwocents Jan 30 '13 at 16:13
  • I had trouble with the "fs -rmr dir" if the directory didn't already exist. However, Pig has the "rmf" command that seems to be working. http://stackoverflow.com/questions/11110403/how-to-force-store-overwrite-to-hdfs-in-pig – mytwocents Jan 30 '13 at 18:41