I'm trying to get to the bottom of an issue where the oom-killer is killing certain jobs off on a server. The server has 48GB of physical ram, with 20GB of swap, and I'm trying to find out what processes are the ones using memory. Output from free ties up with what I'd expect to see on this server, but output from both top and ps show over 600% of memory committed to processes, with a total resident memory footprint of 258GB.
PID USER PR NI VIRT RES SHR S CPU %MEM TIME+ COMMAND
52788 usradm 20 0 47.123g 0.018 0.013t S 0 39.35 36:06.0 tsk_W59
52793 usradm 20 0 41.668g 0.01 0.010t S 0 22.28 44:56.8 tsk_W64
52815 usradm 20 0 41.638g 9.689 9.534g S 0 20.49 10:34.7 tsk_W85
52813 usradm 20 0 41.665g 9.074 8.890g S 0 19.19 48:03.6 tsk_W83
52808 usradm 20 0 41.630g 8.641 8.507g S 0 18.28 18:32.1 tsk_W79
23445 usradm 20 0 41.661g 8.536 8.362g S 0 18.05 02:37.8 tsk_W84
52811 usradm 20 0 41.641g 8.324 8.178g S 2 17.6 49:12.3 tsk_W81
52769 usradm 20 0 41.604g 8.241 8.133g S 0 17.43 22:55.0 tsk_W40
52806 usradm 20 0 41.645g 8.122 7.968g S 0 17.18 23:23.4 tsk_W77
52812 usradm 20 0 41.622g 8.085 7.955g S 0 17.1 45:00.1 tsk_W82
39198 usradm 20 0 41.653g 7.979 7.816g S 0 16.87 26:59.3 tsk_W76
52754 usradm 20 0 41.601g 7.486 7.383g S 0 15.83 16:36.8 tsk_W25
52798 usradm 20 0 41.642g 7.476 7.329g S 0 15.81 26:48.0 tsk_W69
52755 usradm 20 0 41.604g 7.471 7.367g S 0 15.8 12:01.3 tsk_W26
52800 usradm 20 0 41.610g 7.449 7.321g S 0 15.75 52:18.5 tsk_W71
52799 usradm 20 0 41.625g 7.44 7.310g S 0 15.73 27:58.2 tsk_W70
36047 usradm 20 0 41.746g 7.292 7.033g S 0 15.42 16:20.4 tsk_W75
52801 usradm 20 0 41.633g 7.2 7.062g S 0 15.23 27:59.8 tsk_W72
52807 usradm 20 0 41.676g 7.115 6.924g S 1 15.05 53:07.0 tsk_W78
52803 usradm 20 0 41.641g 6.814 6.669g S 0 14.41 27:38.2 tsk_W74
52748 usradm 20 0 41.609g 6.811 6.706g S 0 14.41 08:55.4 tsk_W19
52744 usradm 20 0 41.602g 6.634 6.532g S 0 14.03 06:05.1 tsk_W15
52753 usradm 20 0 41.600g 6.15 6.050g S 0 13.01 12:39.0 tsk_W24
52750 usradm 20 0 41.602g 5.904 5.806g S 0 12.49 09:00.7 tsk_W21
52751 usradm 20 0 41.605g 5.771 5.659g S 0 12.2 05:46.6 tsk_W22
52745 usradm 20 0 41.605g 5.766 5.667g S 0 12.19 06:33.6 tsk_W16
52746 usradm 20 0 41.608g 5.464 5.357g S 0 11.56 07:31.6 tsk_W17
52752 usradm 20 0 41.614g 5.308 5.205g S 0 11.23 09:13.3 tsk_W23
52747 usradm 20 0 41.608g 5.235 5.136g S 0 11.07 15:23.0 tsk_W18
52749 usradm 20 0 41.598g 4.859 4.758g S 0 10.28 07:40.8 tsk_W20
52738 usradm 20 0 41.595g 4.563 4.471g S 0 9.651 04:21.7 tsk_W9
52742 usradm 20 0 41.599g 3.968 3.872g S 0 8.392 03:31.4 tsk_W13
52731 usradm 20 0 41.590g 3.867 3.779g S 0 8.178 02:07.8 tsk_W2
52741 usradm 20 0 41.595g 3.834 3.742g S 0 8.109 05:35.7 tsk_W12
52735 usradm 20 0 41.593g 3.719 3.630g S 0 7.866 03:10.4 tsk_W6
52737 usradm 20 0 41.598g 3.592 3.500g S 0 7.597 05:33.7 tsk_W8
52743 usradm 20 0 41.593g 3.525 3.433g S 0 7.456 04:45.2 tsk_W14
52740 usradm 20 0 41.599g 3.442 3.342g S 0 7.28 05:00.9 tsk_W11
52739 usradm 20 0 41.599g 3.393 3.302g S 0 7.175 03:53.7 tsk_W10
52736 usradm 20 0 41.597g 3.116 3.022g S 0 6.59 03:43.0 tsk_W7
52732 usradm 20 0 41.601g 3.096 3.009g S 0 6.547 02:35.4 tsk_W3
52734 usradm 20 0 41.596g 2.683 2.593g S 0 5.674 02:29.6 tsk_W5
52733 usradm 20 0 41.593g 2.658 2.568g S 0 5.622 02:35.5 tsk_W4
52729 usradm 20 0 41.598g 2.264 2.176g S 0 4.787 04:23.5 tsk_W0
52787 usradm 20 0 41.641g 2.123 1.969g S 0 4.491 15:26.6 tsk_W58
52730 usradm 20 0 41.597g 1.602 1.523g S 0 3.389 02:11.1 tsk_W1
70422 usradm 20 0 41.716g 1.434 1.191g S 0 3.034 23:37.4 tsk_W80
52802 usradm 20 0 41.656g 1.369 1.210g S 0 2.895 54:25.8 tsk_W73
52773 usradm 20 0 41.664g 1.323 1.151g S 0 2.799 12:22.8 tsk_W44
52782 usradm 20 0 41.633g 1.32 1.180g S 0 2.791 29:15.6 tsk_W53
52784 usradm 20 0 41.688g 1.286 1.101g S 0 2.72 09:17.5 tsk_W55
Can someone elaborate on how to pick this apart? I've not had a situation before where memory appears to be over provisioned on a server like this.
TIA
