0

virsh 0.10 I am using qcow2 overlay and a backing file, also qcow2. The overlay is growing way too quick, more than 20 GB in 24 hours, even though I am not writing very much to disk. Is this normal? What causes them to grow? I assume any write occurs there but it seems to be growing way more than a single image would under the same circumstances.

2 Answers2

2

Worth noting that modern filesystems spread writes out across a whole device, so you'll find the overlay grows inline with the amount of writes rather than the amount of data on the disk at any one time.

So writing a 100M file, deleting it and writing it again will actually result in a 200M overlay increase.

Worth checking to see if anything is writing and deleting things. A contemporary suspect might be anything that uses a leveldb database.

John Leach
  • 36
  • 2
0

qcow will grow up to the size specified, and only with new writes to previously untouched blocks. If your guest is deleting and writing files rapidly, it will fill that space out quickly, while the guest will still show plenty of unused space, simply because you also delete files.

You can deduplicate the qcow2 file, simply zero-fill it using sdelete on windows or dd if=/dev/zero of=/some/path/to/file on linux, then delete the zero file and run qemu-img convert on the file - qemu-img will create a new file without the zero filled blocks.

dyasny
  • 18,802
  • 6
  • 49
  • 64
  • The guest runs software builds so it downloads source code every night and removes it then downloads a fresh copy from SVN the next night. This is going to be a problem I guess. How do people deal with this issue? Do I need to routinely dedupe the file with sdelete? What a hassle. –  Oct 20 '13 at 19:40
  • @GreggLeventhal Have you considered using snapshots? Another advantage of that you always have the same starting state for each build. – Lekensteyn Oct 20 '13 at 19:59
  • @Lekensteyn Please elaborate. Do you mean creating a new External snapshot every night and then removing it? The problem with this is from time to time we will install Windows updates which need to be perpetual, I wouldnt want the OS updates to get reverted. –  Oct 20 '13 at 20:18
  • @GreggLeventhal See http://qemu-project.org/Documentation/CreateSnapshot for the most simplistic storage snapshot possible. The idea is to shutdown the machine, remove the current image 2 (which was based on image 1) and then continuing with a new snapshot 3 based on 1. Should you ever need to install updates, update image 1 or create another snapshot based on 1. – Lekensteyn Oct 20 '13 at 20:52
  • The most obvious solution is to preallocate the entire space and be done with it, the snapshot will not be able to outgrow the original image size, so you might as well leave that alone. Moreover, why did you use a snapshot? They aren't recommended for production, especially when heavy IO is used, and compiling definitely does. Adding more snapshots, like @Lekensteyn suggests will only add overhead, waste even more space and complicate the solution even more. – dyasny Oct 20 '13 at 21:29
  • @dyasny Do you have a source for that recommendation against using snapshots? My base images have unused space zeroed out and are compacted. It gets a very limited size (as much as is necessary for operations) and gets thrown away after use. I don't know what size your projects are, but what about creating a ramdisk, throw some gigabytes of RAM at it and write the object files in RAM? That prevents writing to the qcow2 filesystem at all. – Lekensteyn Oct 20 '13 at 21:34
  • The task I am trying to accomplish is to have a copy of the base disk on a second server, if I need a point in time clone, all I have to do is copy the overlay to the second server and apply it to the copy of the base image and I can have a clone without having to copy the data in the base image multiple times. I am using this method because I basically need to be able to revert to a "snapshot" while leaving the current state of the original VM untouched and running. This was the solution I came up with, but the disk space is going to be an issue. –  Oct 20 '13 at 21:39
  • @Lekensteyn sources? VMWare, Red Hat, citrix... it's not a qcow2 issue, snapshots have overhead everywhere. COW is a problematic format, that introduces overhead. As for personal experience, I've been doing high level support and consulting for KVM for 5 years, and for VMware before that, and I've had a chance to see plenty of examples. As for doing things in memory, or on all-SSD arrays for that matter, that's a different story, where overhead becomes moot. I am against running prod systems on a snapshot, if you're doing stateless rollbacks, that's a different thing. – dyasny Oct 21 '13 at 00:18
  • @GreggLeventhal yes, when you are using snapshots, you get to have potential space savings (lots of VMs derived from a single base image), but you usually get hit on IO and single VM space consumption. For PIT copies, I would suggest having a large backup host where you can store snapshots, and pull the needed images out of there if you need to roll back. In production, the less snapshots the better. and BTW, do you need to PIT the entire OS? Build servers are normally preconfigured with something like koji, code is versioned in git/svn/etc and ready builds go into the nightly dirs – dyasny Oct 21 '13 at 00:22
  • It seems like the overlay has stopped growing so quickly once it reached a certain size, but the logical volume it lives on it not full yet. It appears the guest vm is operating properly though, even though the qcow2 isn't growing so fast. Does that make sense? History: I had a 146 GB single qcow2 with a few internal snapshots. I converted it to vmdk and then back to qcow2 which got rid of the snapshots and left me with a 100 GB qcow2. I made that the base image and added an overlay. Now the overlay is about 46GB which puts the total size at where the single VM was before. Significant? –  Oct 22 '13 at 13:42
  • Basically I am wondering if perhaps the overlay is overwriting blocks where deleted files are now that it has reached some sort of limit (as I originally thought it would behave) –  Oct 22 '13 at 13:43
  • I have mentioned that before - the thin provisioned snapshot will not overgrow the original image size. If the base image is 100Gb, that's how large the snapshot can grow (giving you 200Gb of real disk usage). btw, `qemu-img convert` can convert qcow2 to qcow2, you don't need to go to vmdk and back – dyasny Oct 22 '13 at 17:29