Why does the tez engine also add a reduce phase to the simplest insert statement, and how to remove it through configuration?

Question

This is the hive-sql:

insert into my_orc_table_25 select * from my_orc_table limit 5;

And these are the schemas:

CREATE TABLE my_orc_table (
    id INT,
    name STRING
)

STORED AS ORC;

CREATE TABLE my_orc_table_25 as select id,name from my_orc_table limit 25;

My env : hive-3.1.0,tez-0.10.2

I tried to modify the following configuration items, but it didn't work

set hive.compute.query.using.stats=false;
set hive.stats.fetch.column.stats=false;
set hive.stats.fetch.partition.stats=false;
set hive.groupby.skewindata=false;
set hive.exec.dynamic.partition=false;

Please read [How to upload an image to a post?](https://meta.stackexchange.com/questions/75491/how-to-upload-an-image-to-a-post) to improve your _question_. — Eugène Adell, Apr 09 '23 at 18:37

score 0 · Answer 1 · answered Apr 08 '23 at 21:01

0

Reduce stages are required to write data to HDFS.

Map-only jobs read data as-is, and don't further process it.

answered Apr 08 '23 at 21:01

OneCricketeer

179,855
19
132
245

score 0 · Answer 2 · answered Apr 11 '23 at 04:39

0

From explain plan I can see reducer is doing compute stats and file merge.

You can try after disabling both of them using below settings:

set hive.stats.autogather=false;
set hive.merge.tezfiles=false;

answered Apr 11 '23 at 04:39

Raid

188
1
10

Why does the tez engine also add a reduce phase to the simplest insert statement, and how to remove it through configuration?

2 Answers2