2

I have IOT devices (arm64) running Ubuntu with SD cards (formatted as ext4 with Journaling) where my application logging(python logging library) is done to files on that SD card, overall the write speed (as reported by iotop) is around 40KB/s (the device operates 24/7/365)

What I see that once in a while (week or so?) there is a spike in iowait (see attached screenshot from netdata). When this happens my proccess get stuck for 5-15 seconds which is a lot! enter image description here

Now I know that I should change my logging to be non-blocking to avoid my process getting stuck if there is an issue with the disk but it seems excessive this amount of time considering the fact the write speed is very low.

It has gotten worse since I increased logging but still it is not a lot of data.

My next steps are:

  1. Use QueueHandler to do logging without blocking
  2. Disable journaling on sdcard
  3. Disable docker logging as it is also writing to disk.

But I want to understand the underlying issue that causes this kind of stalls, what can it be?

Uriel Katz
  • 187
  • 1
  • 2
  • 10

1 Answers1

0

Not a full solution but adding QueueHandler made my app survive this high loads.

It is easy to simulate this with slowpokefs or just doing a lot of IO (like taring a big folder) while logging constantly.

Uriel Katz
  • 187
  • 1
  • 2
  • 10