1

I'm running puppet and it functions okay, but it's taking about 15 minutes on a sync without any changes, and it seems to me that this shouldn't take more than a minute or two. Here are some stats:

  • ping 40ms
  • 49 files, 3 MB, largest is 90K
  • master is Ubuntu 12.04 (precise) w/ puppet 2.7.11
    • (upgrading to puppet 2.7.23 didn't help)
  • agent is Debian 7/64 w/ puppet version 2.7.23
  • server is serving one client, only

I have tried with and without Apache tunnel/WEBrick/passenger with the same results.

What are some strategies for diagnosing this and and improving on the sync time?

UPDATE: In my case, in addition to the selected answer for using the --evaltrace option, I used strace and found that TCP connections were timing out and that my firewall was rate-limiting. (Why puppet didn't report the timeout and why it’s not reusing the connection, I don’t know.)

danorton
  • 735
  • 1
  • 8
  • 26

2 Answers2

8

The most direct approach is running the agent with the --evaltrace flag like

puppet agent --test --evaltrace

This will give you a good idea of which resources are the most expensive.

For your large files, you may want to experiment with different checksum alternatives.

Felix Frank
  • 3,093
  • 1
  • 16
  • 22
  • Cool! I wasn't aware of that flag. – ewwhite Jul 21 '14 at 16:56
  • 1
    Okay, so why is this so slow? It's only 12Kb. `/Stage[main]/Mailmxbackup/File[mailmxbackup-mailman-aliases.db]: Evaluated in 31.29 seconds` – danorton Jul 21 '14 at 17:00
  • Ugh, no clue. Is the inode of any directory in its path unusually large? (I.e., has the directory grown beyond usual proportions at some point)? That will slow down Linux file operations drastically. – Felix Frank Jul 21 '14 at 17:03
  • Ack! I found it. Shorewall was rate-limiting. I edited the question to remove the unrelated symptoms. – danorton Jul 21 '14 at 19:50
1

Check : https://tickets.puppetlabs.com/browse/PA-3380 which describes slowness caused by a mix of SELinux and the number of mounts.

Nobody is picking this up, which is a shame, as most (enterprise) users will notice a speedup of at least 50%, and much less impact on system load

sysadmin1138
  • 133,124
  • 18
  • 176
  • 300
Frank
  • 11
  • 1