0

On big Linux systems running Centos, we sometimes see messages like

kernel: <program_name>: page allocation failure: order:<N>, mode:0xNNNN

Reading articles like these:

https://utcc.utoronto.ca/~cks/space/blog/linux/DecodingPageAllocFailures

https://utcc.utoronto.ca/~cks/space/blog/linux/WhyPageAllocFailure

it sounds like these messages are mostly spurious warnings resulting from some bad memory check logic in the kernel.

I understand that "order:0" messages are a big deal, but we never get those.

We do get useless higher order messages like the one above, which are just "no action required" spew that clogs up our /var/log/messages file.

Page after page of ignorable stack dumps make it hard to see real errors. This lowers system reliability.

So, is there anyway to redirect just these messages away from the /var/log/messages file?

Ben Slade
  • 149
  • 1
  • 5

1 Answers1

1

"Spurious" or "bad" is not the conclusion Chris made in the blog you linked. They explained how the allocator attempting low order first is not intuitive.

Understand your workload by looking at these:

  • Which user space program
  • The call trace including which kernel subsystems originated the requests
  • Order of the allocation size
  • Free chunks of each order in the memory zones
  • Memory capacity planning, including totals from /proc/meminfo

Provide these details to the support of your distro, or here on Server Fault. Ideas of things to try may include tuning parameters, or a newer kernel.

Like many things logged from Linux, this is printk() based. warn_alloc() in mm/page_alloc.c cannot be suppressed individually, but you can supress all messages with priority lower than LOGLEVEL_WARNING (defined as 4). There is a sysctl for that, so create /etc/sysctl.d/printk.conf with something like

# console_loglevel, default_message_loglevel, minimum_console_loglevel, default_console_loglevel
kernel.printk = 3       4       1       7

Note that this will also suppress a number of other potentially interesting warnings.

Or, you might centralize your logging and alerting, and be able to handle a large number of lines. printk + syslog is more than a person can read, define how to extract signals interesting to you from the noise.

Community
  • 1
John Mahowald
  • 32,050
  • 2
  • 19
  • 34
  • Thanks for the answer. From what I've read, this is sometimes caused by a driver allocating large blocks of kernel memory. It don't the time, expertise, or Linux priv's to debug this. I have looked at /proc/meminfo or zoneinfo, but these are subtle problems involving Intel memory mapping, so it's hard to get a clear picture. – Ben Slade Feb 11 '20 at 16:55
  • If you look back at the history of this, year after year, people are making tuning changes to fight this. IMHO, multiplying the size of the Linux messages file by 100 times with what looks like endless stacktraces is a "bad" thing. There should be an option to turn it off (at least for order>0) or redirect it to another file. – Ben Slade Feb 11 '20 at 17:00
  • You can choose to change the priority that is logged, or to centralize logging to a remote host and not grow the messages file. But reducing these specific messages, no, you would have to post process syslog yourself. – John Mahowald Feb 12 '20 at 16:57