2

I have enabled SElinux in enforcing mode on Amazon Linux and see that the time taken for file access (read/write/update) has increased by an average of 2-4 seconds. The source process is Salt and the file access is related to the file.managed state of Salt https://docs.saltstack.com/en/latest/ref/states/all/salt.states.file.html#salt.states.file.managed There are no corresponding denials in the audit.log file. As a brute-force approach, we tried adding allow rules for all denials in the log using audit2allow but could not improve the time taken to access files.

Is SElinux known to cause a performance hit in filesystem access? Are there known processes on a linux system which affect time to access files?

sce
  • 23
  • 3

2 Answers2

3

Short answer: yes, it cause added latency, but it is so small you generally do not care

Long answer: SELinux add some latency because:

  • its label is stored inside an extended attribute, which is an additional metadata tag (to be read/parsed) attached to each file;

  • it must compare the just-read metadata to the currently loaded binary policy.

To minimized the performance impact, the linux kernel keep an AVC cache to bypass the most heavy steps in the read/compare process. More information can be found here and here

The takeaway is that on the general case, SELinux impacts performance by 0-2% only, and can be ignored from a performance standpoint

shodanshok
  • 47,711
  • 7
  • 111
  • 180
2

SELinux is routinely enforcing on systems that do more IOPS than yours. Also, IOs regularly taking multiple seconds is intolerably poor performance, no matter what storage system or additional overhead.

Something else is happening. Use Linux's rich performance tools to reveal it. Some starting ideas:

  • Measure the utilization of all performance resources on the host. CPU, disk bandwith, disk IOPS, memory, and so on.
  • Get a flame graph of what is on CPU when this thing is slow.
  • Use a programming language specific profiler on the problem lines of code. In this case, salt's file functions.
  • Get the block IO latency distribution such as with biolatency.
  • Benchmark raw IOs on this storage system using tools like fio, or just touch

(Some of these require recent kernels for bpf and other features. I don't know how good Amazon Linux's tooling is for all of these.)

John Mahowald
  • 32,050
  • 2
  • 19
  • 34
  • Based on @Michael Hampton's comment, we dug deeper and found an issue with our Salt code. The files being accessed had a "requires" condition, and the evaluation of that condition was causing the delay. Once fixed, the file access time with selinux running became identical to when selinux was not running. – sce Aug 06 '19 at 06:10
  • The details of this slow condition would be interesting as an answer if you are willing to share. That's the actual solution, our answers are just how it probably isn't SELinux and the tools to check. – John Mahowald Aug 06 '19 at 10:52