Greenplum - Out of memory

Question

When trying to query from gpdb cluster. getting Out of memory error with error code 53400.

System Related information

TOTAL RAM =30G
SWAP =15G
gp_vmem_protect_limit=8192MB
TOTAL segment = 8 Primary, 8 mirror = 16
SEGMENT HOST=2

Getting error :

ERROR: Out of memory (seg2 slice109 datanode01:40002 pid=21691)
SQL state: 53400
Detail: VM protect failed to allocate 8388608 bytes from system, VM Protect 4161 MB available

We tried

gpconfig -c gp_vmem_protect_limit -v 4114 vm.overcommit_ratio = 95

Then, getting this error. P ERROR: XX000: Canceling query because of high VMEM usage. Used: 3704MB, available 410MB, red zone: 3702MB

Also , getting this symptom

Prod=# show runaway_detector_activation_percent;
 runaway_detector_activation_percent 
-------------------------------------
 90
(1 row)

Please suggest what could be the setting in this case. Also, What is the root cause of OOM error?

Any help on it would be much appreciated?

What is the query? How much RAM in the system? How many nodes, how many segments per segment server? — A. Scherbaum, Sep 03 '17 at 15:35
Please refer this link https://discuss.pivotal.io/hc/en-us/articles/205180467-How-to-Identify-Out-of-Memory-OOM-Errors — Anuraag Veerapaneni, Sep 04 '17 at 09:29
Thanks for support doc. as per doc OS is reporting OOM here should i set gp_vmem_protect_limit is set to low? Currently getting following error : ERROR: Out of memory (seg2 slice109 sungpmsh01:40002 pid=21691) DETAIL: VM protect failed to allocate 8388608 bytes from system, VM Protect 4161 MB available — vs0731, Sep 05 '17 at 10:51
@ A. Scherbaum query - is with clause CTE expression. RAM - 30 GB Number of nodes - 2 data node - one primary node , one standby node , Total Segment Host = 16 , segment host is 2 — vs0731, Sep 05 '17 at 11:12
gpconfig -c gp_vmem_protect_limit -v 4114 vm.overcommit_ratio = 95, In this statemtn, gp_vmem_protect_limit should not be 4114. Calculate your server parameters from this link http://greenplum.org/calc/ — Anuraag Veerapaneni, Sep 05 '17 at 14:11
@AnuraagVeerapaneni Thanks for help, I have recalculated using the same calc site. gpconfig -c gp_vmem_protect_limit -v 2700 and implemented but still it is showing the Same error: OOM — vs0731, Sep 05 '17 at 14:23
Is there any change in values in the error message. Can you paste the error? — Anuraag Veerapaneni, Sep 05 '17 at 14:26
Now , error look likes ERROR: 53400: Out of memory (seg3 slice181 datanode01:40003 pid=23454) — vs0731, Sep 05 '17 at 14:29
@AnuraagVeerapaneni is it possible to discuss this over chat window? — vs0731, Sep 05 '17 at 14:50
Can you send detail message also? Like Detail: VM protect failed to allocate 8388608 bytes from system, VM Protect 4161 MB available. is there a chat window in stackoverflow? — Anuraag Veerapaneni, Sep 05 '17 at 15:01
ERROR: 53400: Out of memory (seg3 slice181 datanode01:40003 pid=23454) - This is complete error we are getting — vs0731, Sep 05 '17 at 15:05
Also, when I am using vmprotect limit 2700 with Overcommit ratio 95 . error got changed and statment mem =1 GB GETTING THIS ERROR ERROR: XX000: Canceling query because of high VMEM usage. Used: 2431MB, available 269MB, red zone: 2430MB (runaway_cleaner.c:135) (seg2 slice42 datanode01:40002 pid=24032) (cdbdisp.c:1320) — vs0731, Sep 05 '17 at 15:43
Out of memory error is not coming now. but getting this error ERROR: XX000: Canceling query because of high VMEM usage. Used: 2431MB, available 269MB, red zone: 2430MB (runaway_cleaner.c:135) (seg2 slice42 datanode01:40002 pid=24032) (cdbdisp.c:1320) — vs0731, Sep 05 '17 at 16:01
@AnuraagVeerapaneni error changed ERROR: XX000: Canceling query because of high VMEM — vs0731, Sep 05 '17 at 18:39

Greenplum - Out of memory

0 Answers0