I need your help.
I have the DRBD-cluster (9.6.0, kernel 3.10.0-957.21.3, CentOS 7). In this cluster I have two drbd-disks:
- drbd0 for SSD
- drbd1 for HDD
With drbd0 (SSD, sda) all good - it's in UpDate status. But with drbd1 (HDD, sdb) I see next:
# drbdadm status
drbd0 role:Primary
disk:UpToDate
slave role:Secondary
peer-disk:UpToDate
drbd1 role:Primary
disk:UpToDate
slave role:Secondary
replication:SyncSource peer-disk:Inconsistent done:0.17
It's VERY slow - I get 0.17% in 6 hour. I known, what HDD more slowly then SSD, but this vary bad.
Info:
I have this configuration (commented lines - it's my experiments):
# cat /etc/drbd.d/global_common.conf
global {
usage-count yes;
}
common {
net {
protocol B;
}
}
# cat /etc/drbd.d/drbd0.res
resource drbd0 {
on master {
device /dev/drbd0;
disk /dev/mapper/vg_ssd_drbd-lv_ssd_drbd;
meta-disk internal;
address 192.168.100.15:7788;
}
on slave {
device /dev/drbd0;
disk /dev/mapper/vg_ssd_drbd-lv_ssd_drbd;
meta-disk internal;
address 192.168.100.17:7788;
}
net {
sndbuf-size 10M;
rcvbuf-size 10M;
ping-int 2;
ping-timeout 2;
connect-int 2;
timeout 5;
ko-count 5;
max-buffers 128k;
max-epoch-size 8192;
verify-alg md5;
}
disk {
c-plan-ahead 20;
c-min-rate 1M;
c-max-rate 600M;
c-fill-target 2M;
al-extents 3389;
}
}
# cat /etc/drbd.d/drbd1.res
resource drbd1 {
on master {
device /dev/drbd1;
disk /dev/mapper/vg_hdd_drbd-lv_hdd_drbd;
meta-disk internal;
address 192.168.100.15:7789;
}
on slave {
device /dev/drbd1;
disk /dev/mapper/vg_hdd_drbd-lv_hdd_drbd;
meta-disk internal;
address 192.168.100.17:7789;
}
net {
#sndbuf-size 1M;
#rcvbuf-size 1M;
ping-int 2;
ping-timeout 2;
connect-int 2;
timeout 5;
ko-count 5;
#max-buffers 12k;
#max-epoch-size 8192;
#verify-alg md5;
}
disk {
#c-plan-ahead 20;
c-min-rate 1K;
c-max-rate 600M;
#c-fill-target 2M;
al-extents 919;
}
}
Servers have 10Gbps channel link-to-link - both locate in one room.
I can show you my monitoring:
At nigth I sync my SSD - all good. But at day I tried to sync my HDD and it's very cry.
IO to growing instantly, but read and write operations to disk not have no one servers. With NET-traffic situation is identical.
If I connecting to server, I see this picture:
top - 12:52:35 up 1 day, 10:44, 1 user, load average: 1.01, 1.06, 1.26
Tasks: 492 total, 1 running, 491 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 0.0 us, 0.3 sy, 0.0 ni, 0.0 id, 99.7 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
DRBD1 (I see him in iostat) loading iowait in CPU on 100%, but WriteKB and ReadKB is ~zero.
I googled it's problem and I was advised show TCP Buffer, but it's fine. I flush all DRBD-setting for drbd1 to default, but it's don't have result.
I have try to diagnostic problem by yourself and finded two anomaly:
One. I see some one "Time Spent Doing I/Os" in 1 sec. I think, what I get timeout here.
Two. In me HDD mounted catalog I see big difference for df/du and ls commands. Maybe this is feature KVM, but I not sure.
du -sh /data/hdd-drbd/*
170M /data/hdd-drbd/awx-add.qcow2
7.7G /data/hdd-drbd/awx.qcow2
2.0G /data/hdd-drbd/template-DISABLE.qcow2
ls -lah /data/hdd-drbd/
total 9.8G
drwxr-xr-x 2 root root 74 Aug 16 17:37 .
drwxr-xr-x. 8 root root 91 Aug 14 22:11 ..
-rw------- 1 qemu qemu 201G Aug 15 19:41 awx-add.qcow2
-rw------- 1 qemu qemu 7.7G Aug 18 17:26 awx.qcow2
-rw------- 1 root root 46G Aug 15 13:48 template-DISABLE.qcow2
Now I will going to move all data to SSD disk and will try resync empty disk - maybe - it's will be fine. But I need your help for this problem - do you have some ideas for this situations?
EDIT:
One more - why I resyncing my storages? I added to some PV in my LVM for drbd[0/1] and resize drbd-devices. Maybe this is important information... Before this operations drbd worked fine.
EDIT2:
The empty disk resync selfsame...
