2

What I have done

I have built a Docker Swarm cluster where I am running containers that have persistent data. To allow the container to move to another host in the event of failure I need resilient shared storage across the swarm. After looking into the various options I have implemented the following:

  1. Installed a Ceph Storage Cluster across all nodes of the Swarm and create a RADOS Block Device (RBD). http://docs.ceph.com/docs/master/start/quick-ceph-deploy/

  2. Installed Rexray on each node and configure it to use the RBD created above. https://rexray.readthedocs.io/en/latest/user-guide/storage-providers/ceph/

  3. Deploy a Docker stack that mounts a volume using the rexray driver e.g.

    version: '3'
services:
  test-volume:
    image: ubuntu
    volumes:
      - test-volume:/test
volumes:
  test-volume:
    driver: rexray

This solution is working in that I can deploy a stack, simulate a failure on the node that is running then observe the stack restarted on another node with no loss of persistent data.

However, I cannot mount a rexray volume in more than one container. My reason for doing is to use a short lived "backup container" that simply tars the volume to a snapshot backup while the container is still running.

My Question

Can I mount my rexray volumes into a second container?

The second container only needs read access so it can tar the volume to a snapshot backup while keeping the first container running.

0xF2
  • 314
  • 3
  • 17
BenCaldwell
  • 323
  • 1
  • 15

2 Answers2

1

Unfortunately the answer is no, in this use case rexray volumes cannot be mounted into a second container. Some information below will hopefully assist anyone heading down a similar path:

  1. Rexray does not support multiple mounts:

    Today REX-Ray was designed to actually ensure safety among many hosts that could potentially have access to the same host. This means that it forcefully restricts a single volume to only be available to one host at a time. (https://github.com/rexray/rexray/issues/343#issuecomment-198568291)

  2. But Rexray does support a feature called pre-emption where:

    ..if a second host does request the volume that he is able to forcefully detach it from the original host first, and then bring it to himself. This would simulate a power-off operation of a host attached to a volume where all bits in memory on original host that have not been flushed down is lost. This would support the Swarm use case with a host that fails, and a container trying to be re-scheduled. (https://github.com/rexray/rexray/issues/343#issuecomment-198568291)

  3. However, pre-emption is not supported by the Ceph RBD. (https://rexray.readthedocs.io/en/stable/user-guide/servers/libstorage/#preemption)

BenCaldwell
  • 323
  • 1
  • 15
1

You could of course have a container that attaches the volume, and then exports it via nfs on a dedicated swarm network, the client containers could then access it via nfs

krad
  • 1,321
  • 1
  • 15
  • 12
  • This would work but when I posted this I was looking at how to avoid the bottleneck and single point of failure of the host with the volume. – BenCaldwell Jan 29 '20 at 03:33