6

in our company we have a lot of kvm clients on multiple server, most of the clients are running ubuntu 16.04 as well as their host systems.

The filesystem of choice became EXT4, for clients and hosts. Recently I used some of the awesome snapshot features provided by BTRFS to setup a backup server with incremental backups.

Some research provided the clue one does never use BTRFS for the KVM host, because the fragmentation of the FS slows the clients down until they finally even freeze.

Is there any recommendation / do / don't for using BTRFS on the KVM client?

We are reconsidering our FS choices for clients and hosts, are there advantages using XFS over EXT4 (client/host or one sided only)?

chrstnwhlrt
  • 325
  • 6
  • 15

4 Answers4

4

On google you can find a lot of website who talk about the performance of differents filesystem with KVM.

Take a look at this one : ZFS, BTRFS, XFS, EXT4 and LVM with KVM – a storage performance comparison

According to the author Gionatan Danti :

The tested scenarios are:

  1. Qcow2 backend on top of XFS filesystem on top of a raw MD device. Both thin and partial (metadata only) preallocation modes were benchmarked;
  1. Logical Volumes backend, both in classical LVM (fat preallocation) and thin (thin lvm target) modes. Moreover, thin lvm was analized with both zeroing on and off;
  1. raw images on XFS and EXT4 on top of classical LVM, relaying on filesystem sparse-file support for thin provisioning;
  1. raw images on XFS and EXT4 on top of thin LVM, relaying on thin lvm target for thin provisioning. In this case, LVM zeroing was disabled as the to-be-zero blocks are directly managed inside the filesystem structures;
  1. raw images BTRFS on top of its mirror+stripe implementation (no MD here). I benchmarked BTRFS with CoW both enabled and disabled (nodatacow mount option)
  1. raw images ZFS on top of its mirror+stripe implementation (no MD again)

He conclude by :

For VMs storage, stay well away from BTRFS: not only it is marked a “Tech Preview” from RedHat (read: not 100% production ready), but it is very slow when used as a VM images store.

Another blog talk about BTRFS, you can read on a lot of forums that Copy On Write (COW) need to be disable for getting better performances with KVM.

Chris Irwin talk about the benefits of BTRFS and talk about an alternative :

There are other tools, or you could roll your own cron-job. So what about ZFS? I thought ZFS did all these things?

Yes, it does Why not just use ZFS?

Go ahead

link : live with btrfs

Another way to know if it's okay for your use is testing by yourself if the performance is good and if it is reliable without copy on write.

If BTRFS is not the best for you, you can try ZFS. You have the same Backup fonctionality and lot of other improvements but it a bit tricky to implement in linux.

RomanSt
  • 1,207
  • 1
  • 15
  • 32
P0pR0cK5
  • 330
  • 1
  • 6
  • Please include the gist of these links in your answer. If the give links go dead some day your answer becomes useless. – Gerald Schneider Jun 23 '17 at 09:08
  • How to do this ? – P0pR0cK5 Jun 23 '17 at 09:11
  • See [answer] and [How to reference material written by others](https://serverfault.com/help/referencing). – Gerald Schneider Jun 23 '17 at 09:14
  • It's not worth much what a single person like me says, but I have seen phenomenal performance from raw block devices on LVM. With a proper RAID backend, I could easily get GBs per second in the guest VMs - Win, Linux and MacOS. And you still retain the ability to take snapshots. – Alexander Gräf Jun 21 '22 at 22:09
2

My KVM orchestration solution of choice, oVirt, uses LVM volumes handed as raw disks to VMs for maximum performance, scalability and flexibility. You can do both qcow2 and LVM snapshots. If you are building a new storage solution and want to try something SDS-ish and fancy, you could go with Ceph and RBD-access to volumes instead.

dyasny
  • 18,802
  • 6
  • 49
  • 64
  • The main issue with LVM based snapshots is that they can be painfully slow to write on. The reason for this is not incompetent programmers, but how normal file systems and snapshots work. This is why all the fuzz about COW file systems came into life, because there snapshots are cheap and fast. – Marc Stürmer Jul 07 '17 at 20:02
  • This is why I said you can use both qcow and lvm snaps – dyasny Jul 07 '17 at 21:02
  • 1
    @MarcStürmer While standard, classical LVM volumes have slow snapshots, they can be made faster by using larger chunk size (default: 4KB). For example, you can run something similar to `lvcreate -s --name -c 64k`. The really fast snapshots are, however, the one obtained via the new [lvmthin](http://man7.org/linux/man-pages/man7/lvmthin.7.html) tech, with use CoW internally. – shodanshok Jul 07 '17 at 21:34
1

Take a look at the official Btrfs Gotchas site on their own wiki: https://btrfs.wiki.kernel.org/index.php/Gotchas

Especially this point:

Fragmentation

Files with a lot of random writes can become heavily fragmented (10000+ extents) causing thrashing on HDDs and excessive multi-second spikes of CPU load on systems with an SSD or large amount a RAM. On servers and workstations this affects databases and virtual machine images. The nodatacow mount option may be of use here, with associated gotchas.

So this alone speaks against it, what use is a COW file system if the one feature you want to use it for is not being useable with virtual host images in a fast manner at all? If you want to use a COW filesystem, then take ZFS.

Also keep in mind, that Coreos moved away from Btrfs to EXT4 as default file system because it was still too buggy back then too years ago.

https://www.phoronix.com/scan.php?page=news_item&px=CoreOS-Btrfs-To-EXT4-OverlayFS

So while Ext4 might not be so flashy and Mr. Fancy Pants, it's a trusted and reliable work horse. If you are looking for a file system aside Ext4 on Linux and unwilling to use ZOL/switching to FreeBSD, maybe XFS is worth giving a try.

Please note that though I am using Btrfs on my home desktop with Gentoo and quite recent kernels, every few months it has still some quite nasty hiccups which make the system unbootable and needs to be repaired manually, which takes time for investigating on Google and using trial&error, and if it doesn't work, a proper backup.

The real question you should be asking yourself is this: why should we move away from Ext4? It seems to work perfectly well for your use case. Just because you read something about a new, flashy thing doesn't mean that you really need it. Think about it.

Marc Stürmer
  • 1,904
  • 13
  • 15
1

CentOS abandon the support of btrfs ref: https://www.theregister.co.uk/2017/08/16/red_hat_banishes_btrfs_from_rhel/ so for CentOS btrfs is not a smart choice.

btrfs is seen as the next FS standard, but ZFS gain a lot of attention too, mostly by Ubuntu.

Most of warning against btrfs are old and or concern specific configuration like RAID6 (which mean more than 2drives)

openSUSE choose btrfs and support it well. such as an example on the /var/lib/libvirt directory they disable the COW so they obviously face some issue and fix them.

I'm running openSUSE with btrfs on 2 SSD build as a RAID1 which is use as root where I host 5 VM (KVM) and 20 containers (Docker).

I'm quite satisfied with it.

JOduMonT
  • 129
  • 3