0

I have a PHP application running in Docker on two environments with the same Docker version, Docker container, OS (CentOS 7) and kernel. On one, I've discovered performance is about 5x worse. Using perf, the system call sk_run_filter appears to be the primary culprit for the performance. I know this is called by seccomp, but this is also enabled and using the default profile on both servers, so known seccomp performance concerns don't seem like the cause of this.

The only difference I was able to find between these environment was the Docker filesystem—the slow environment is using overlay instead of overlay2. We will be upgrading this in hopes that it resolves the issue, but looking to understand if this is the source of these slow sk_run_filter calls or if something else might be the cause.

To get to the question: what would cause poor sk_run_filter performance with Docker, and could it be the overlay filesystem?

Marco Bonelli
  • 63,369
  • 21
  • 118
  • 128
user116518
  • 1
  • 1
  • I don't recommend running in production with this, because seccomp is an important component of Docker's security, but can you try it with `--security-opt=seccomp:unconfined` to see if it's seccomp or the filesystem? – Nick ODell Dec 11 '21 at 21:17
  • @NickODell Unfortunately, the slower version of this is the production environment—it's also in Docker Swarm so I don't have the tooling to turn seccomp off. – user116518 Dec 11 '21 at 21:27
  • Hmmm. Well, if the kernel versions weren't the same, I would suggest that [constant action bitmaps for seccomp](https://lwn.net/Articles/834785/) might be the cause of the difference. Any way you can post [flame graphs](https://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html) for both systems? – Nick ODell Dec 11 '21 at 21:41
  • 1
    Haven't used those before - will have to get that figured out and come back. Thanks for taking a look at this. After running some strace checks, it looks like the underlying problem might be the clock source—the slow environment is making system calls which appear to be cratering performance. – user116518 Dec 11 '21 at 23:36

0 Answers0