0

Can I modify the value of a partitioned table, just by changing the name of the partition directory?

The table I now have has year and month as partitions. The values were stored as decimal so the partitions are "2016.0" instead of just "2016" and "3.0" instead of "3".

Can I just rename the directories and have the values in the partitions updated?

markwusinich
  • 108
  • 7
  • So I was looking to run a test, but I don't see any hdfs commands to rename a directory. So this might be moot. – markwusinich Jul 15 '16 at 21:27
  • I was hoping to do something like this `hadoop fs -rename /dev/.../year=2016.0/month=4.0 /dev/.../year=2016.0/month=4 ` but that does not work. – markwusinich Jul 15 '16 at 21:38
  • I tried this: `> hadoop fs -mv /dev/.../year=2016.0/month=4.0 /dev/.../year=2016.0/month=4` and it worked, in that it moved the directory, but now queries against the table no longer work, because hive `Caused by: java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File hdfs://prodmid/dev/.../year=2016.0/month=4.0 does not exist.` – markwusinich Jul 15 '16 at 21:44
  • mv ing it back made it so that it worked again. – markwusinich Jul 15 '16 at 21:46

1 Answers1

3

First rename the directories:

hadoop fs -mv /dev/year=2016.0 /dev/year=2016
hadoop fs -mv /dev/year=2016/month=4.0 /dev/year=2016/month=4

Let the hive metastore know about new location/partition:

ALTER TABLE logs PARTITION(year = 2014, month = 4) 
SET LOCATION 'hdfs://dev/year=2016/month=4';
Ronak Patel
  • 3,819
  • 1
  • 16
  • 29