vmstat output incompatible

Question

as you can see when run a vmstat like below.

bash-3.00# vmstat 1 5
kthr      memory            page                     disk              faults        cpu
r  b  w   swap   free    re   mf   pi   po   fr   de sr m1 m1 m1 m2   in   sy   cs    us sy id
2  0  0 12812792 7094520 668  1023 213  939  939  0  0  1  1  1  0  30519 66084 35693 12 7  81
10 19 0 5469784  797392  265  325  583  1119 1119 0  0  1  1  0  0  35404 75227 44916 14 8  78
34 10 0 5577672  846056  570  554  3802 1233 1233 0  0  1  0  0  0  35013 72253 46505 10 10 79
19 0  0 5576352  843112  462  2825 1200 963  963  0  0  0  0  0  0  35070 73182 44379 13 8  78
2  0  0 5582624  854664  1698 1885 134  1198 1198 0  0  2  1  1  0  35688 71013 43705 11 7  82

what is wrong with this ? because the memory usage is about %95.

***---****---****----*****

Since the system is an HA system here is the output for standby machine.

bash-3.00# vmstat 1 5
kthr      memory            page          disk                faults            cpu
r b w   swap   free    re   mf   pi   po fr de sr m1 m1 m1 m2 in    sy    cs    us sy id
1 0 0 13949104 8516736 553  1047 121  21 20 0  0  1  1  1  9  30691 69172 34352 12 7  81
0 0 0 12456136 7171232 1122 2554 4117 0  0  0  0  3  1  2  0  31505 85240 35958 11 8  81
0 0 0 12353888 7147488 1401 664  0    0  0  0  0  0  0  0  0  31647 73507 35881 13 6  81
0 0 0 12345536 7129280 13   40   0    0  0  0  0  0  0  0  0  30787 69357 34695 14 6  80
0 0 0 12345440 7125040 217  1646 0    24 24 0  0  0  0  0  0  31390 73710 38393 12 10 78

nope it is not. because it is in an HA system and the hot standby machine output : `bash-3.00# vmstat 1 5 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr m1 m1 m1 m2 in sy cs us sy id 1 0 0 13949104 8516736 553 1047 121 21 20 0 0 1 1 1 9 30691 69172 34352 12 7 81 0 0 0 12456136 7171232 1122 2554 4117 0 0 0 0 3 1 2 0 31505 85240 35958 11 8 81 0 0 0 12353888 7147488 1401 664 0 0 0 0 0 0 0 0 0 31647 73507 35881 13 6 81 0 0 0 12345536 7129280 13 40 0 0 0 0 0 0 0 0 0 30787 69357 34695 14 6 80` — Olgun Kaya, May 21 '12 at 06:23
sorry the output can not be formatted as well as the initial question! — Olgun Kaya, May 21 '12 at 06:24
one more comment : if you check both outputs you will see that; after second period the problemmatic one's free memory decreases so much. where it is not on the hot standby one. — Olgun Kaya, May 21 '12 at 06:27
Olgun: You can edit your question and add further output to it. That way you can format it. Concerning the drop in vmstat output: I believe this to be related to the startup of vmstat itself, I see this on nearly every run of vmstat and I tend to ignore the first couple of lines. — Bram, May 21 '12 at 06:43
@OlgunKaya I meant please explain *why* this is not what you expect. — mgorven, May 21 '12 at 06:50
Can you be very precise about what your concern is? The machine seems to have very little memory free, which is exactly what you want. (Free memory is forever wasted. It's not like if you use 1GB less today you can use 1GB more tomorrow.) — David Schwartz, May 21 '12 at 12:15

ramruma · Accepted Answer · 2012-05-22T04:22:59.513

Solaris? Which version (presumably 8 or 9 from the idle column)? Sun cluster or vcs?

Anyway, ignore the first line since that is an average since boot.

Is a backup running on the passive node, by the way, and causing the minor faults and page-ins on a machine ostensibly doing nothing? And look at the context switches.

On the active node, memory usage at 90 per cent probably is just memory used by whatever you are running: Oracle's SGA, perhaps; and the rest is in use for cacheing. This is normal. The system does seem to be io-bound, at least for the few seconds you show. There is paging in and out but no serious memory pressure, so this probably reflects memory-mapped io. Some swap space is in use but that may reflect pre-allocation rather than any shortfall.

Run: vmstat -s; vmstat -p 10 5. If it is Solaris, also: prstat -a; prstat -t; prstat -a -s rss; and on Solaris 9, echo ::memstat | mdb -k (which can take a while so do not panic). These will show what the system is using memory for, and which applications are using it. If you think your application software is leaking memory, then compare measurements from prstat over time.

If you are consistently io-bound, then run iostat -xn 60 and speak to your storage administrators if appropriate. But that is a separate matter.

thanks. using some of these commands we found that some sendmail apps stacked ( ~140 ) each using 20 MB of memory. When we restarted the service it seems now the system working as expected load level. standby machine memory level : 54% working machine memory level : 82% which is acceptable — Olgun Kaya, May 23 '12 at 07:42

score 1 · Answer 2 · answered May 21 '12 at 06:48

I'm not sure what's being asked here but you do remember that the first line of the vmstat output shows the average values for the items since the last reboot ? Subsequent lines show the values for the last sampling period, so in general ignore the first line.

vmstat output incompatible

2 Answers2