Optimizing Disk I/O when running Redis and PostgreSQL on a single machine

Question

Background:

I have a live Django app that utilizes 4 Redis instances.

The first two are big in size: back ups amount to ~2GB and ~4.4GB respectively. The other two are small: ~85M and ~15M.

redis-server --version yields Redis server v=4.0.2 sha=00000000:0 malloc=jemalloc-4.0.3 bits=64 build=401ce53d7b0383ca.

The problem:

It's a busy server running PostgreSQL 9.6.5 as well. PG data and Redis backups are both saved in the secondary drive xvdb.

I've noticed that whenever my big Redis instances start backing up, disk I/O naturally spikes and PostgreSQL commit statements start piling up in the slow log. Behold:

21:49:26.171 UTC [44861] ubuntu@myapp LOG:  duration: 3063.262 ms  statement: COMMIT
21:49:26.171 UTC [44890] ubuntu@myapp LOG:  duration: 748.307 ms  statement: COMMIT
21:49:26.171 UTC [44882] ubuntu@myapp LOG:  duration: 1497.461 ms  statement: COMMIT
21:49:26.171 UTC [44893] ubuntu@myapp LOG:  duration: 655.063 ms  statement: COMMIT
21:49:26.171 UTC [44894] ubuntu@myapp LOG:  duration: 559.743 ms  statement: COMMIT
21:49:26.172 UTC [44883] ubuntu@myapp LOG:  duration: 1415.733 ms  statement: COMMIT

As a consequence, this is how my PostgreSQL commits look like every day:

The question:

Is there anything I can do on the Redis side to help smoothe out this spikey situation? I'd like Redis and PostgreSQL to live in as much harmony as they possibly can on a single machine.

More information:

Ask for more information if you need it.

Machine specs:

AWS EC2 m4.4xlarge (16 cores, 64GB RAM)
Elastic Block Store gp2 volumes (105 IOPS, burst upto 3000 IOPS)

The following config exists in the Append Only Mode section of my Redis conf files:

appendonly no
appendfilename "appendonly.aof"
# appendfsync always
appendfsync everysec
# appendfsync no
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble no

Typical iostat -xmt 3 values are:

10/15/2017 08:28:35 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10.44    0.00    0.93    0.15    0.06   88.43

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvda              0.00     0.00    0.00    2.00     0.00     0.04    38.67     0.00    0.00    0.00    0.00   0.00   0.00
xvdb              0.00     2.67    0.00   44.67     0.00     0.41    18.99     0.13    2.81    0.00    2.81   1.07   4.80

Compare that to the same around the time slow commits are logged:

10/15/2017 10:18:11 PM

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.16    0.00    0.65   11.90    0.04   79.24

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
xvda              0.00     4.00    0.00    1.00     0.00     0.02    48.00     0.00    1.33    0.00    1.33   1.33   0.13
xvdb              0.00     0.00    1.67 1312.00     0.01   163.50   254.90   142.56  107.64   25.60  107.75   0.76 100.00

The first Redis instance has the following snapshotting config:

save 7200 1
#save 300 10
#save 60 10000

The second Redis instance has the following snapshotting config:

save 21600 10
#save 300 10
#save 60 10000

score -1 · Answer 1 · answered Oct 22 '17 at 10:44

-1

I can suggest one solution - docker. docker has the capability to limit the os resources that a particular container will use. In your case, the os resource seems to be the disk i/o.

The --device-read-bps flag limits the read rate (bytes per second) from a device. For example, this command creates a container and limits the read rate to 1mb per second from /dev/sda:

$ docker run -it --device-read-bps /dev/sda:1mb

docker also has a flag --device-read-iops=[] which limit read rate (IO per second) from a device.

Your image here will be redis, but it won't hurt to move postgres also to docker. This will help you limit the disk i/o rates on redis containers so that postgres has more of it.

This solution should work for you but the infrastructure changes might be a little painful to achieve.

You can refer this SO post for docker limit i/o speed

answered Oct 22 '17 at 10:44

Arpit Goyal

2,212
11
31

This is my only realistic choice given the scenario? – Hassan Baig Oct 23 '17 at 02:34
you can wait for other answers... meanwhile I will try to figure out if it is possible at os level. – Arpit Goyal Oct 23 '17 at 04:38
@HassanBaig did you go through this link? `https://unix.stackexchange.com/questions/48138/how-to-throttle-per-process-i-o-to-a-max-limit` – Arpit Goyal Oct 23 '17 at 05:38
That my friend is a useful link as well. Thanks for suggesting it. – Hassan Baig Oct 23 '17 at 09:42
if u found it helpful do upvote, accept the solution. – Arpit Goyal Oct 23 '17 at 09:44
Yep, I'm going to try your docker solution in a while, probably will accept if everything goes A-OK by the morning. Native I/O throttling and/or using a separate host are also options that are on the table. I'll get back to this. – Hassan Baig Oct 23 '17 at 09:47

Optimizing Disk I/O when running Redis and PostgreSQL on a single machine

1 Answers1