Hive compactions not triggered automatically - HDP_2.6.5

Question

I created a table in hive to test compaction,

CREATE TABLE part_test(id int, name string, city string) PARTITIONED BY (dept string)  clustered by (city) into 5 buckets stored as orc TBLPROPERTIES('transactional'='true');

In the screenshot we can see that there are many delta directories created(around 12 files).

The compaction is automatically initiated but it is not running or working to compact the delta directories.The following are the properties we set in hive.

set hive.compactor.initiator.on=true;
set hive.compactor.worker.threads=1;
set hive.compactor.delta.pct.threshold=0.1f;
set hive.compactor.delta.num.threshold=10;
set hive.exec.dynamic.partition=true;  
set hive.exec.dynamic.partition.mode=nonstrict; 
set hive.enforce.bucketing=true;

In the screenshot 2 we can see the compaction status as initiated but not executed for more than a day. How does we make the compaction to work ? do we need any other properties to set ?

score 1 · Accepted Answer · answered Dec 03 '19 at 14:37

1

Can you check your hive site configuration, the property of 'Number of threads used by Compactor'

If it is 0 then change it to greater than 0 (maybe 2). Then check for compactions status by using 'show compactions;'.

answered Dec 03 '19 at 14:37

Saravanakumar

93
6

Hive compactions not triggered automatically - HDP_2.6.5

1 Answers1