0

Can I configure the amount of memory available to all HAWQ segment instances and the amount of memory available for each segment?

Chunling Wang
  • 21
  • 6

6 Answers6

1

This is covered in the system requirements.

Jon Roberts
  • 2,068
  • 1
  • 9
  • 11
0

In Apache Hawq, virtual segments are used as containers of the executors. As a result, the memory used by queries is controlled by the number of virtual segments. You can use GUC hawq_rm_memory_limit_perseg to control the total memory size of each host(segment instance) and to control the memory size of a virtual segment, you can create your own resqueue with specified memsize for each container(256M as default).

huan zhang
  • 11
  • 1
0

The virtual segment memory usage is set by hawq_rm_stmt_vseg_memory, which calculates the total memory for all forked QE. Since different query statements may resides on the same vseg, so hawq_rm_stmt_vseg_memory will be shared across different queries.

ztao1987
  • 71
  • 7
  • But I saw the comment of hawq code, "hawq_rm_stmt_vseg_memory" refers to "the memory quota of one virtual segment for one statement." Is it shared across different queries? And should "hawq_rm_stmt_vseg_memory * vitrual segments number per segment * total query statements" be no larger than "hawq_rm_memory_limit_perseg"? – Chunling Wang Jul 07 '16 at 10:08
  • I think hawq_rm_stmt_vseg_memory means the memory quota of one vseg for all statements. By default, it's 256M. So in theory, hawq_rm_stmt_vseg_memory * vitrual segments number per segment should be no larger than hawq_rm_memory_limit_perseg. Can other guys confirm this? – ztao1987 Jul 08 '16 at 01:58
0

hawq_rm_stmt_vseg_memory is the memory quota(size) of one virtual segment, the default value is 128mb. That means, the memory size of one virtual segment is 128mb, one query may request many virtual segments.

Wen Lin
  • 21
  • 2
0

@Wen Lin, you mentioned "one query may request many virtual segments", so one virtual segment is also shared by many queries, right? Then all shared queries will share the memory quota, which is 128mb by default?

ztao1987
  • 71
  • 7
0

@ztao, virtual segments are not shared by queries. For one query, it asks virtual segments from RM, and return virtual segments to RM when it finishes.

huan zhang
  • 11
  • 1