0

We're upgrading our Symfony 3.4 app (LAMP w/ opcache/php-fpm) to flex (as per https://symfony.com/doc/current/setup/flex.html). In load testing, we're seeing increased CPU usage, particularly system CPU.

Changes we've noticed in profiling:

  • Increased CPU interrupts.
  • perf top shows far more cpu time spent in the mutex_spin_on_owner and osq_lock kernel calls. From research, this suggests we've got a bottleneck on access to some shared resource.
  • sar -n SOCK shows more UDP ports being used, and indeed tcpdump output shows an ~ten-fold increase in DNS lookups sent. The additional lookups seem to include nameserver lookups for client IPs; however, we've not added any related logging or code to do this. We're comparing the same codebase, with exception of the changes required for the upgrade to flex (which granted, are significant, but primarily configuration changes).

Things we're thinking are ruled out:

  • Memory swapping - this hasn't seemed to change via vmstat.
  • We've profiled io, cpu and mem for php-fpm, mysqld, redis-server, httpd via pidstat (-r, -u, -d flags) and don't think we're seeing anything significant. Similarly, screen recordings of top don't show any perceptible issues/differences.
  • We're using a CDN for static assets. We've started cache-breaking images, but have confirmed this is working as expected (not serving many requests for assets from our server at all).
  • We use mariadb with persistent connections. Confirmed no change in connections/s, so don't think we're waiting on db access.

So, looking for any thoughts on what we may have misconfigured, or how to trace what is initiating these additional DNS lookups.

Matt
  • 1
  • 2

1 Answers1

0

Well, we're not 100% sure why yet, but doing the following resolved the issue for us: Removing the symfony/web-link package installation, and removing/commenting the preload config in webpack_encore.yaml:

webpack_encore:
    ...
    # preload all rendered script and link tags automatically via the http2 Link
    # header
    preload: true

Will update with the specific cause when/if we can.

Matt
  • 1
  • 2