1

I'm checking the disk speed of a hdd on CentOS 6 using dd command. The setup is a DRBD setup - with this server being primary. To ensure that I get correct values, I execute the dd command 3 times with different output files and then take average of the read/write time. However the first reading of the dd command is order of magnitude slower than the next 2. Eg.

time -p dd if=/dev/zero of=/mailstore/testfile bs=16k count=16384
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 88.5175 s,
3.0 MB/s
real 90.12
user 0.00
sys 0.66

time -p dd if=/dev/zero of=/mailstore/testfile1 bs=16k count=16384
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 0.226015 s, 1.2 GB/s
real 0.30
user 0.00
sys 0.22


time -p dd if=/dev/zero of=/mailstore/testfile2 bs=16k count=16384
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 0.22094 s, 1.2 GB/s
real 0.22
user 0.00
sys 0.21

Is this normal ? Should I ignore the first reading and take 3 more after that ?

As suggested by poige, the dd command gives consistent output using the opflag=direct option. Eg.

time -p dd if=/dev/zero of=/mailstore/filetest33 bs=16k count=16384 oflag=direct
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 296.587 s, 905 kB/s
real 296.61
user 0.03
sys 1.07

time -p dd if=/dev/zero of=/mailstore/filetest44 bs=16k count=16384 oflag=direct
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 260.308 s, 1.0 MB/s
real 260.42
user 0.04
sys 1.13

time -p dd if=/dev/zero of=/mailstore/filetest56 bs=16k count=16384 oflag=direct
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 253.681 s, 1.1 MB/s
real 253.68
user 0.03
sys 1.06
amolkul
  • 111
  • 5
  • 9

2 Answers2

5

dd has direct option which requires kernel to bypass any caching and send data directly to block device. If you need benchmarking your device, not RAM, you have to use it, for e. g.:

dd if=/dev/zero of=direct_output bs=1M count=100 oflag=direct

poige
  • 9,448
  • 2
  • 25
  • 52
  • Does it mean that first time caching did not happen but second and third time it happened ? – amolkul Jan 20 '14 at 09:01
  • @amol, it means you're measuring incorrectly, I don't want to guess. Use `direct` then we'll see. – poige Jan 20 '14 at 09:55
  • Isn't the `sync` command worth it as well? – SamK Jan 20 '14 at 13:50
  • @SamK, not too much — why blow the caches anyway if you don't need'em. – poige Jan 20 '14 at 14:19
  • sync doesn't blow away the caches; it forces any pending writes to be finished and committed to storage. If anything, it'd be good to sync before he does do any measurements to make sure he's not going to block on unrelated writes. This 'cat /proc/meminfo | grep Dirty' should show if there's pending writes. To drop the read buffer cache, which admittedly isn't relevant to this, use, as root, "echo 1 > /proc/sys/vm/drop_caches" – etherfish Jan 20 '14 at 14:42
  • @etherfish, nobody except you said `sync` blows away the caches. `dd` w/o direct would do blow them. amen. – poige Jan 20 '14 at 15:02
  • Sorry, I misinterpreted your statement, "@SamK, not too much — why blow the caches anyway if you don't need'em." as that. Also, I just realized SamK must have been referring to the dd flag and I was thinking of /bin/sync. amen? Hallelujah. – etherfish Jan 20 '14 at 15:06
  • 1
    dd with oflag=direct is safe to use on production servers right ? – amolkul Jan 23 '14 at 09:24
  • @amol, made my day :) – poige Jan 23 '14 at 10:11
0

So, you said it was mounted over the network. Which filesystem, CIFS or NFS? Either way, I suspect you may've been delegated the file. When your client gets a file delegated, it's able to cache writes locally. However, when you use O_DIRECT (that's what oflag=direct means), the writes are sent to the server immediately instead of any caching.

Either way, there's something strange going on. You should be getting more than 1.1MB/s unless you're actually using just 10Mbps ethernet.

Also, you can get a mid-transfer update for speed by running, in another window, killall -USR1 dd If you check the dd man page, you'll see that the USR1 signal does not kill/stop dd but print out I/O statistics. I often do this when waiting for large disk transfers:

while sleep 10; do killall -USR1 dd; done

Oh, and if you suspect disk caching, use this command to flush the readcache: echo 1 > /proc/sys/vm/drop_caches

Good luck!

etherfish
  • 1,757
  • 10
  • 12