0

I ordered a new server because I have to move to higher performance server. However, this new server is totally awful. I tried to get the benchmark by using fio here is the result.

[root@s206716 ~]# fio --filename=/var/lib/mysql/fio.tmp --size=1m --bs=512 --name=go --rw=write --fdatasync=1
go: (g=0): rw=write, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=psync, iodepth=1
fio-3.19
Starting 1 process
^Cbs: 1 (f=1): [W(1)][39.3%][w=3KiB/s][w=7 IOPS][eta 01m:08s]   
fio: terminating on signal 2

go: (groupid=0, jobs=1): err= 0: pid=108048: Thu Apr 20 18:32:15 2023
  write: IOPS=18, BW=9401B/s (9401B/s)(397KiB/43242msec); 0 zone resets
    clat (usec): min=5, max=126657, avg=951.49, stdev=9446.81
     lat (usec): min=5, max=126658, avg=951.94, stdev=9446.80
    clat percentiles (usec):
     |  1.00th=[     6],  5.00th=[     6], 10.00th=[     9], 20.00th=[    10],
     | 30.00th=[    13], 40.00th=[    14], 50.00th=[    20], 60.00th=[    22],
     | 70.00th=[    23], 80.00th=[    25], 90.00th=[   157], 95.00th=[   297],
     | 99.00th=[  8094], 99.50th=[102237], 99.90th=[126354], 99.95th=[126354],
     | 99.99th=[126354]
   bw (  KiB/s): min=    2, max=  118, per=100.00%, avg= 9.21, stdev=20.45, samples=86
   iops        : min=    4, max=  236, avg=18.43, stdev=40.89, samples=86
  lat (usec)   : 10=24.43%, 20=30.23%, 50=32.75%, 250=6.55%, 500=3.02%
  lat (usec)   : 750=0.63%, 1000=0.76%
  lat (msec)   : 2=0.25%, 4=0.13%, 10=0.38%, 100=0.38%, 250=0.50%
  fsync/fdatasync/sync_file_range:
    sync (msec): min=2, max=233, avg=53.50, stdev=65.67
    sync percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    5], 40.00th=[    5], 50.00th=[    5], 60.00th=[   68],
     | 70.00th=[   97], 80.00th=[  107], 90.00th=[  132], 95.00th=[  199],
     | 99.00th=[  226], 99.50th=[  228], 99.90th=[  234], 99.95th=[  234],
     | 99.99th=[  234]
  cpu          : usr=0.02%, sys=0.09%, ctx=2445, majf=0, minf=17
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,794,0,0 short=794,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=9401B/s (9401B/s), 9401B/s-9401B/s (9401B/s-9401B/s), io=397KiB (407kB), run=43242-43242msec

Disk stats (read/write):
    dm-0: ios=104/2979, merge=0/0, ticks=971/141634, in_queue=142605, util=96.36%, aggrios=104/2973, aggrmerge=0/22, aggrticks=974/143011, aggrin_queue=143985, aggrutil=95.40%
  sda: ios=104/2973, merge=0/22, ticks=974/143011, in_queue=143985, util=95.40%

I have no idea why the speed process is really slow. This one is not a production server. I have just importing all the files and testing out. So, there is no traffic yet. I have asked the provider to do hardware testing and they say that there is no issue.

What should I do with this thing? I have done many things. Such as trying to move to another server I have (low spec server) and it has no issue. However, using this high performance server is even lowest than others.

  • 2
    Please edit your question to add: OS distro version (CentOS?), which file system type, the rated performance of the block device (SSD model, storage array model, promised IOPS). And also add other fio option combinations to find significant differences via scientific method. --fdatasync is brutal to file systems, for example. – John Mahowald Apr 21 '23 at 13:58
  • Additional DB information request from new server, please. OS, Version? RAM size, # cores, any SSD or NVME devices on MySQL Host server? Post TEXT data on justpaste.it and share the links. From your SSH login root, Text results of: A) SELECT COUNT(*), sum(data_length), sum(index_length), sum(data_free) FROM information_schema.tables; B) SHOW GLOBAL STATUS; after TESTING and 4 hours UPTIME C) SHOW GLOBAL VARIABLES; D) SHOW FULL PROCESSLIST; E) STATUS; not SHOW STATUS, just STATUS; G) SHOW ENGINE INNODB STATUS; for server workload tuning analysis to provide suggestions. – Wilson Hauck Apr 29 '23 at 20:55
  • Let `fio` run for longer. Add more parallelism - try `--numjobs=16`, `--iodepth=64`, increase data size (such as `--size=10G`). Do you still get same results? It is possible server is using some fancy network-attached storage which is quick but may have higher latency, so `fdatasync=1` is what's killing test results. Real databases don't behave quite this way - they try to do at least some batching. – rvs May 02 '23 at 19:52
  • 1
    Oh, and another thing - `--bs=512` will not be representative on modern software & hardware. Try at least 4k. With 512 byte writes you are guaranteed each of them becomes read-modify-write as SSDs use much larger block sizes. – rvs May 02 '23 at 19:55
  • @James-Hartford additional consideration for rvs May 2 comment, MySQL basic unit of work is always 16384 - so use bs=16384 to be paired with MySQL typical operations. – Wilson Hauck May 16 '23 at 11:01

0 Answers0