2

In our cluster, we have minimum container size as 8 GB most of the hive queries use 1 container. ( but surely may not use all the memory allocated ) some of the spark jobs just use 2GB or 4GB we don't use that much memory for all our queries as per observation. still all containers are used up.

So, is there anyway we can manage effectively.

We have total of 30 vcores total of 275 GB of memory as we have to allocate 1 vcore per container, that bottles to 30 containers

is there a way i can efficiently leverage all 8gb of container? or increase container number or do some other things. any suggestions will help

Kumar
  • 119
  • 10
  • When you say minimum container size, are you referring to YARN as is `yarn.scheduler.minimum-allocation-mb`? – tk421 Mar 19 '19 at 16:46
  • so, that one is set to 8gb because we have 30 vcores, can i set it lower than 8gb in that case, can that container work in essence . please bear with me as i am newbie here. if each container need 1 core then max i can use are 30 containers. is that correct understanding – Kumar Mar 19 '19 at 16:52
  • 1
    Not sure about `Spark`, but I know for `Tez`: `Divide RAM/CORES = Max TEZ Container size` So in my case: `128/32 = 4GB`. `2-4GB` on `YARN` is usually better than making huge containers... – Petro Mar 19 '19 at 18:16
  • ok thanks, so if we run multiple TEZ queries via one hql file, will they all run in one container ? (although may have different DAG id but same appid) – Kumar Mar 19 '19 at 18:27
  • 1
    They'll run one after the other in order, so 1 container. `set hive.prewarm.enabled=true;` and `hive.prewarm.numcontainers` to get reusable containers, and submit separately to get parallel (depending on YARN queue setting of course) – Petro Mar 19 '19 at 18:40
  • yeah thanks, as i have limited containers, my goal is to ensure that each container is used optimally, so we try running many queries in one file to ensure utilization is optimal. – Kumar Mar 19 '19 at 18:44
  • Hi @Kumar so you will have 30 containers and therefore 30 nodes on your Hadoop cluster. That means you will have only one vcore for each node which is running the node manager and the spark executor? Is that correct? – abiratsis Mar 21 '19 at 15:11
  • no, 30 containers on 1 node, which means parallelism degree of 30 on 1 node, we have many nodes, so u can can multiple 30* number of nodes . however, not all queries need 8 gb container, this means its huge wastage of memory per container. – Kumar Mar 22 '19 at 12:21

0 Answers0