How to set the resources for different hive stages from the EXPLAIN?

Question

Is there any way to provide resources basing on the stages from the EXPLAIN query.

What do you mean by *"provide resources"* -- specify how many containers to use for each step? how much RAM / cores per container? both? — Samson Scharfrichter, Jan 14 '16 at 19:27
@Samson yes, I am referring to the containers and the associated rRAM/cores. — wandermonk, Jan 15 '16 at 03:18

score 3 · Answer 1 · edited May 23 '17 at 12:07

AFAIK the only things you can set are:

number of R / exact count set mapred.reduce.tasks=** (as long as it makes sense)
number of R / max count set hive.exec.reducers.max=** + hint for computing the actual number set hive.exec.reducers.bytes.per.reducer=*****
resources per M container (when using MR) / set mapreduce.map.memory.mb=** + set mapreduce.map.java.opts=-XMX** <~80% of total for heap>
resources per R container (when using MR) / set mapreduce.reduce.memory.mb=** + set mapreduce.reduce.java.opts=-XMX** <~80% of total for heap>
resources per container (when using TEZ) / set hive.tez.container.size=** + set hive.tez.java.opts=-XMX** <~80% of total for heap>

Note that if your query shows 3 Map steps and 2 Reduce steps, the settings will apply to all steps...

1 Answers1