0

I have looked through many duplicate questions over the past few hours on here but none really shed light on the issues (confidence) I have been faced with in Work, not exactly.

So we have a new Hetzner machine. 4 x 3TB, 64GB RAM, LSI MegaRAID SAS 9260-4i (Just had a BBU fitted also) and a Xeon E3-1275, Also some good network connections. Its perfect for our use case.

The Problem

I am the Sysadmin / Linux Guy, 90% of things I am fine with, but I rarely build servers from scratch and all our other servers use Software Raid (mdadm). I have never set up Hardware Raid from scratch with megacli, I have done this now but any feedback is appreciated, Other than using ext2, ext3, ext4 and btrfs I have no experience with what to expect with Xfs or ZFS

What I would like advice with

  1. Because Raid5 gives us more space than Raid10, boss wants to opt for Raid5. I am not sure if Raid10 would make much difference anyway as all files are served over the internet to mostly UK users (UK -> Germany). Do you think Raid5 to Raid 10 will make much performance
    difference?

  2. My boss has requested that we use xfs as the filesystem, I am partial to this, We will not be generating so many files, We are just looking for a filesystem that is more like something a NAS would use to store files until our update every 2 hours as we send files out to clients, we will also be writing a fair amount of data and using quite a few IOPs at some stages of the day. We will sometimes have developers connecting to the servers (via website) to test their new software release. For the all purpose use really, I was planning with just ext4 or ext3, But If you think xfs or even ZFS would be better, I would love to learn anyway.

    Any suggestions?

  3. LVM, Now this is something I personally wanted to add to all our new server builds. Snapshots and volume resizing has saved our @$$ so many times on 2 of the servers we have, I thought it would be a good idea to use it as part of a standard build. I have only ever used this with ext4 filesystems though, Should this matter? I am guessing not?.

I know this is a long very specific story, but I would really appreciate any help you can give or even if you can point me in the right direction, I have been reading about these subjects on various different boards, articles since Monday and I can now tell my boss is getting frustrated with me :(

root# lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda            8:0    0  8.2T  0 disk 
├─sda1         8:1    0  512M  0 part /boot
├─sda2         8:2    0  8.2T  0 part 
│ ├─vg0-root 253:0    0  8.1T  0 lvm  /
│ ├─vg0-swap 253:1    0   64G  0 lvm  [SWAP]
│ └─vg0-tmp  253:2    0   20G  0 lvm  /tmp
└─sda3         8:3    0    1M  0 part 

root# megasasctl
a0       LSI MegaRAID SAS 9260-4i encl:1 ldrv:1  batt:FAULT, module missing, pack missing, charge failed
a0d0      8382GiB RAID 5   1x4  optimal
a0e252s0   2794GiB  a0d0  online  
a0e252s1   2794GiB  a0d0  online  
a0e252s2   2794GiB  a0d0  online  
a0e252s3   2794GiB  a0d0  online


root# megacli -LDInfo -Lall -aAll | grep 'Cache Policy:'
Default Cache Policy: WriteBack, ReadAhead, Direct, Write Cache OK if Bad BBU
Current Cache Policy: WriteBack, ReadAhead, Direct, Write Cache OK if Bad BBU

root# df -Th
Filesystem           Type      Size  Used Avail Use% Mounted on
udev                 devtmpfs   16G     0   16G   0% /dev
tmpfs                tmpfs     3.2G  600K  3.2G   1% /run
/dev/mapper/vg0-root xfs       8.2T   11G  8.1T   1% /
tmpfs                tmpfs      16G     0   16G   0% /dev/shm
tmpfs                tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs                tmpfs      16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/vg0-tmp  reiserfs   20G   33M   20G   1% /tmp
/dev/sda1            ext4      488M   52M  401M  12% /boot
tmpfs                tmpfs     3.2G     0  3.2G   0% /run/user/0

Thank You in advance for any possible help.

hexce
  • 3
  • 2
  • 2
    PLEASE PLEASE PLEASE do not use RAID 5 with those disks, it's considered staggeringly dangerous to use these days, especially with larger/slower disks - I can bore you with why the maths work out that way or you can look around on this site and others to see how many people have lost all their data because of it. Only R6/60 and R1/10 (and RAID-Z with ZFS) are considered safe these days. XFS is perfectly fine for most use cases, and LVM, although it does eat up a little performance, is - in my opinion - worth the effort, especially if you're not totally experienced and may need to make changes. – Chopper3 Nov 14 '19 at 12:05
  • Also that's a VERY old CPU - 8+ years old - hardly new...bit worried about that tbh – Chopper3 Nov 14 '19 at 12:07
  • 1
    @Chopper3 Really appreciate the comments, exactly what I was after, Yes I know the CPU is old (my boss is cheap, what can I say) We are replacing a server that had an old i3 in though, so it should be a little better. Its the data I'm more concerned about being "liable" – hexce Nov 14 '19 at 12:28
  • R1/10 still expose you to full data loss in case of double disk failure (m1/m2). Less chance than with R5 where rebuild windows is much longer. The conclusion is that in both cases you need valid data backups. Once you're confident with backup/restore process, you can go with R5. – Chaoxiang N Nov 14 '19 at 13:02
  • 1
    @Chopper3 ChaoxiangN Just to update you both and I appreciate both your answers, I have opted for RAID10, XFS, LVM and created separate partitions for /, /var, /home (/home is where most of our current data is going to reside). The RAID 10 option was because of the mirroring and write performance. Boss suddenly has new use case for the server. We consistently backup off-site also. I really appreciate your answers :) – hexce Nov 14 '19 at 20:05
  • Good work there :) – Chopper3 Nov 15 '19 at 09:11

1 Answers1

0

1/ r5 vs r10 perf : depends on your IO workload (ratio read/write, random/sequential access pattern, workers count in //, io size, read iops, write iops).

From the question 2, I understand that your workload is far from being heavy, and mostly with reads iops, which is a good point when you engage raid5 protection. So I think that your boss is right with r5 (he's the boss, he's always right )

2/ Currently XFS Filesystems can't be shrunk. LVM+ext4 is a good option that I've seen on thousands nodes running critical production stuff. ZFS is also a very good option, especially for snap features, send/receive for disaster recovery, but is not always supported by distros. Depends if you need official support. (maybe canonical provides zfs support on ubuntu, to be verified)

3/ To my mind, LVM is definitely a must-have (except if you choose ZFS) for the reasons you have explained.

PS1 : I would recommend to shrink (mean reinstall due to xfs filesystem...) the root lv to something like 100GB, and create another lvs for data/applications. So as to avoid an app filesystem full make your overall system full.

PS2 : don't you have an issue with your raid controller battery ?

Chaoxiang N
  • 1,283
  • 5
  • 11