2

Through iostat I could find spikes in disk writes once per minute. I think these spikes are caused by fsync, since MongoDB by default flush data to disk every 60 seconds. I could also find out that coinciding with the spikes slow queries are appearing in the slow query log.

There is an option called storage.syncPeriodSecs, but according to the docs:

Do not set this value on production systems. In almost every situation, you should use the default setting.

Executing fsync more often could reduce the spikes, so I would like to know the risks of changing this value in production.

I would appreciate any thoughts.

Details:

  • MongoDB version: 3.2.16

  • Storage engine: WiredTiger

  • Slow queries during the spike: A couple of them, of around 1second. Not stalling the server

  • Deployment: Sharded cluster. Replica sets with two members (primary + secondary)

  • Specs: CPU 8 cores, Memory 64GB, SSD disk

Antonio Val
  • 3,200
  • 1
  • 14
  • 27
  • Please post more details: 1) are you using WiredTiger or MMAPv1 storage engine, 2) how slow are the slow queries, and does it involve "stalling" of the whole server, 3) is this a standalone deployment or a replica set, 4) if a replica set, is this happening on the primary or the secondaries, 5) your MongoDB version, 6) how the hardware is provisioned (RAM, cpu, etc.) – kevinadi Sep 13 '18 at 03:53
  • Thanks for your comment!, a general answer about `storage.syncPeriodSecs` risks was ok, but I edited my question with the details :) – Antonio Val Sep 13 '18 at 04:25
  • @AntonioVal were you able to fix this? – best wishes Oct 13 '20 at 05:22
  • @bestwishes finally it was solved improving the specs, but that's not really a fix. – Antonio Val Oct 14 '20 at 06:18

1 Answers1

1

The WiredTiger storage engine performs a checkpoint every 60 seconds. It sounds like your deployment is struggling during these checkpoint events. This behaviour you're seeing is typical if you're doing a heavy write load into a hardware that is (could be) under-provisioned.

It is usually not recommended to change the syncPeriodSecs value in a production environment since the default value is deemed to be the right balance between memory usage, the number of fsync events, the potential of losing data between fsync events in a crash, and other considerations in a typical hardware configuration.

Changing this value could make the stalls worse. You can of course do some experiment by changing it (lower or higher) to see if it can "smooth out" the fsync events. Having said that, this is an advanced tuning mechanism that is best reserved when other options are not available anymore. If possible, stalls like these can usually be solved by provisioning better hardware, since it appears that the current hardware is struggling under the load you're expecting it to handle.

kevinadi
  • 13,365
  • 3
  • 33
  • 49