21

I have a large (100gb+) database that I'm trying to run with the official postgres image.

I can't store the data in a docker volume because the ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2 file in Docker for Mac has a size limitation of about 60gb.

I'm hesitant to mount a host directory as a volume because file access in mounted host directory volumes is much slower than regular volumes.

These are some useful links that go into more detail on these issues:

  • This discusses the size limitation of the Docker.qcow2 file
  • This also discusses the size limitation of the Docker.qcow2 file
  • This discusses the mounted host directory volume speed issue
  • This gives a nice description of how to replace the Docker.qcow2 file with a file that can grow larger
  • This discusses how the Docker.qcow2 file doesn't shrink as its contents are removed (this isn't directly related, but can further complicate the problem)

Do you all just eat the speed loss and mount a host directory? Do you manually create a qcow2 file that can grow larger with qemu (if you do this, do you need to maintain this file between upgrades)? Do you do something else to handle this issue?

James Chevalier
  • 10,604
  • 5
  • 48
  • 74

3 Answers3

2

The following script delete and re-create a new shrinked Docker.qcow2 file preserving the images passed as arguments.

https://blog.mrtrustor.net/post/clean-docker-for-mac/

Hope this helps.

obe6
  • 1,743
  • 15
  • 23
2

Docker for Mac 18.06 switches from the qcow2 file format to the raw file format, which improves speed and disk usage. The core issue still persists, in that Docker has a limited amount of space that it can use for all of its data. However, you can now set what that limit is within Preferences -> Disk -> Disk image size.

Docker for Mac version 17.12 was the first version to introduce the raw file format, but there were some bugs in the initial release that caused them to remove it as a feature temporarily. You can search that page for 'raw' to look back through the feature's history.

There's a great note here about how Docker for Mac reports its disk usage...

This will display the logical size:

ls -alh ~/Library/Containers/com.docker.docker/Data/vms/0/Docker.raw

While this will display the physical size:

du -h ~/Library/Containers/com.docker.docker/Data/vms/0/Docker.raw

James Chevalier
  • 10,604
  • 5
  • 48
  • 74
1

For a situation like this, I would definitely recommend creating a qcow image that can grow larger.

It is a relatively straightforward process, and you get the performance benefits that are generally quite necessary when running a large database.

programmerq
  • 6,262
  • 25
  • 40
  • 3
    It would not hurt to put some link to how to do that. – Stefano Borini Dec 06 '16 at 11:18
  • 1
    I included a link on how to resize the Docker.qcow2 file in my question: https://forums.docker.com/t/no-space-left-on-device-error/10894/26 ... That site's UI is funky, though - you have to scroll down to beccari's post on Aug 31 that starts with "Resize the image after it was created won't work" – James Chevalier Dec 12 '16 at 14:59