3

Currently I am building a Docker Swarm cluster. It consists of 3 managers and 3 workers. The app I will be deploying on this setup consists of a laravel backend that needs its code in multiple containers for scalability. I already tried GlusterFS volumes and rex-ray with a Ceph backend as shared storage for my volumes. GlusterFS is not reliable and Ceph is in my case a bit overkill and too damn difficult :)

The current setup looks something like this. I have a Percona cluster outside of docker, I run GlusterFS on these servers and simply mount them into the Docker Workers.

                                                Docker Managers
+-------------------------------------------------------------+
|                                                             |
|   +---------+   +---------+   +---------+    +---------+    |
|   |         |   |         |   |         |    |         |    |
|   | HAproxy +---+ HAproxy +---+ HAproxy +----+   SSL   |    |
|   |         |   |         |   |         |    | Manager |    |
|   +----+----+   +----+----+   +----+----+    +---------+    |
|        |             |             |                        |
+-------------------------------------------------------------+
         |             |             |
         |             |             |           Docker Workers
+-------------------------------------------------------------+
|        |             |             |                        |
|   +----+-------------+-------------+--------------------+   |
|   |                                                     |   |
|   |                      Applicaties                    |   |
|   |                                                     |   |
|   +---+--------------+---------------+--------------+---+   |
|       |              |               |              |       |
|       |              |               |              |       |
|   +---+----+     +---+----+     +----+---+     +----+---+   |
|   | Mysql  |     | Mysql  |     | Mysql  |     | Mysql  |   |
|   |   LB   +-----+   LB   +-----+   LB   +-----+   LB   |   |
|   +---+----+     +----+---+     +----+---+     +----+---+   |
|       |               |              |              |       |
|       +---------------+-------+------+--------------+       |
|       |                       |                     |       |
+-------------------------------------------------------------+
        |                       |                     |
        |                       |                     |
        |                       |                     |
+-------+--------+     +--------+-------+    +--------+-------+
|                |     |                |    |                |
|    MySQL01     |     |    MySQL02     |    |    MySQL03     |
|    Gluster01   +-----+    Gluster02   +----+    Gluster03   |
|                |     |                |    |                |
+----------------+     +----------------+    +----------------+

Then I mount them into the php containers like this:

--mount type=bind,source=/mnt/client-data,target=/var/www/html/

This works but this is very slow. The page loading times are around 10 seconds, when the files are not mounted (They exist in the container) the page loading times are around 2-3 seconds.

I came around Flocker and this seems very interesting, but I think a flocker volume can only be mounted on one container. Is this true?

Another solution that I am trying out right now, is that the code gets pulled from git every time a new container gets created. This is actually a good solution but it takes around 5 minutes to pull the code and run composer and when I push updates I need to restart all containers.

What would be the best solution to share my code amongst different containers on multiple hosts? (or even datacenters) Currently I have access to many different storage backends (Ceph, NFS, gluster) and creating a new one is no issue.

EDIT: Why is gluster unreliable in this case? I might have said it wrong above... Gluster is unreliable when using docker volumes on Gluster, with a volume driver plugin. The volumes get mounted nicely most of the time when creating a service, but in case rescheduling happens in the swarm the volumes rarely gets mounted again. I will look into this later to determine what goes wrong, but I don't have a lot of free time at the moment. Gluster is in fact, solid. Just not in this case.

user3942918
  • 25,539
  • 11
  • 55
  • 67
Odyssee
  • 2,393
  • 2
  • 19
  • 38
  • It would be interesting for readers to understand how GlusterFS is unreliable. – user239558 Mar 01 '17 at 20:11
  • I will explain why it is unreliable in this situation later this day, first I have to get my work done :) But it has something to do with creating docker volumes in Gluster, this fails about 50% of the time. – Odyssee Mar 02 '17 at 14:12

2 Answers2

3

What I eventually ended up doing is use CephFS and mount this on the docker hosts. The write speeds are OK, maybe even excellent when compared to glusterfs. The reads are extremely fast on CephFS, this was also the case with GlusterFS.

This setup suits my needs, but I can't use shared docker volumes accross hosts. I have to bind the mounted filesystem in docker containers. Although this is not the end of the world, I am still looking for a convenient and fast way to have my volumes accross all the hosts in the swarm.

Currently I am trying to build a Docker volume plugin for CephFS to achieve my requirements. I'll keep this updated in case someone is interested.

Odyssee
  • 2,393
  • 2
  • 19
  • 38
  • Just out of curiosity, can't you mount your CephFS to `/var/lib/docker/volumes`? It seems like `metadata.db` does not have any host specific content. Anything else that could potentially break if this folder is shared? – inta Apr 23 '22 at 14:35
  • @inta This was 5 years ago, a lot has changed since then. I remember having issues with mounting `/var/lib/docker/volumes` for many reasons. Not sure how it will behave now. – Odyssee Apr 25 '22 at 14:04
0

Note that I have not tried the following.

Since your code is mostly read-only, it seems to me that you "just" need a file system cache. CacheFS for example running on top of NFS.

Since GlusterFS can be mounted as NFS, you should be able to combine the two.

user239558
  • 6,964
  • 1
  • 28
  • 35