0

I understood you can limit Hadoop services via cgroups in Static Service pools. I would like to limit the Hue service, because sometimes, it eats up all the memory we have on the Edge node and hurts our loading processes.
However I wasnt able to find Hue in the static service pool configuration - it only gives me options - HDFS, Impala, YARN, Hbase.
Can the Hue setting be done here, or I would need to do it somewhere else?

Thank you.

k_mishap
  • 451
  • 2
  • 8
  • 17

2 Answers2

0

You can activate cgroup memory options (Cgroup Memory Soft Limit, Cgroup Memory Hard Limit) as you want.

jedijs
  • 563
  • 5
  • 14
0

In this case, you can try to set cgroup memory soft limit in Cloudera Manager Hue configuration page, but I do not believe it will help much. It is a known issue in Hue due to Python memory fragmentation. There are a few common operations in Hue that might trigger it, such as download large query result set (more than 10M) or use HDFS file browser to browse an HDFS directory with a large number of files(1000+). Ask your users to refrain from doing these operations.

If this memory problem keeps happening, you can use the script at https://github.com/cloudera/hue/blob/master/tools/ops/hue_mem_cron.sh to setup a cron job. The script monitors the Hue process memory usage and kill it if it uses too much. You need to configure Cloudera Manager to restart Hue automatically.

Of course, killing the Hue is not an ideal solution. What you can do is to setup Hue HA with a load balancer in front of multiple Hue instances to alleviate the problem. You can follow documentation https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_hag_hue_config.html to set it up

Lan
  • 6,470
  • 3
  • 26
  • 37
  • Awesome, thanks for the loads of information. Im talking with my users about this issues (donwload, browsing) often, but there is always somebody who forges. I will setup the cron as a temporary solution and propose the HA as a correct approach. – k_mishap Sep 10 '17 at 14:49