Running Docker Compose on Azure Batch Shipyard

Question

I was wondering if anyone had any idea how to run compose on Batch Shipyard, or short of directly using compose, allowing multiple containers to work in concert on the same node in a job. Is this possible?

To clarify - I have several containers that work together parse and process a file. The main container utilizes other services via network calls. A simplified example of the compose file that I want to replicate is like so:

version: "3"
services:
  primary:
    image: "primaryimage"
    depends_on:
      - secondary
    environment:
      SECONDARY_URL: secondary:8888
  secondary:
    image: secondaryimage

Within code run in primary there are calls to the URL given in SECONDARY_URL to perform some transformations on the data, and a response is returned.

score 2 · Accepted Answer · answered Feb 19 '18 at 12:22

2

Azure Batch (Shipyard) does not have out-of-the-box support for Docker Compose.

But if you are already familiar with Docker Compose then it's not too hard convert it to shipyard configuration files.

If you want to run MPI/multi-instance tasks (a cluster of nodes cooperating on solving parts of a computation) then take a look at this.

However, Service Fabric does support Docker Compose. So if Docker Compose support is a strict requirement you could combine you Azure Batch setup with calls to a Service Fabric cluster.

answered Feb 19 '18 at 12:22

rasmusgude

489
2
6

To translate compose to a shipyard config, would I consider "tasks" in a job configuration to be equivalent to services in compose? How do containers communicate with each other on the same host? – jimjkelly Feb 19 '18 at 12:28
Yes, that sounds like a fair assumption. When you say 'communicate with each other', do you then mean how they coordinate when a task has finished? for that purpose the "coordination command" can still be used (https://learn.microsoft.com/en-us/azure/batch/batch-mpi#coordination-command). Otherwise the container can communicate using 'regular' means like shared storage (Azure File Storage, RemoteFS) or networking (Azure VNet). You configure this in the Pool configuration. – rasmusgude Feb 19 '18 at 12:51
I edited my question with a little more context. Essentially the communication I'm referring to is communication between a couple of containers working together to achieve a single job/task – jimjkelly Feb 19 '18 at 13:01
Thanks for the clarification. There is no way to make tasks dependant on each other and do name resolving like you can in docker compose. A workaround could be to have 'secondary' write its name and ip to shared storage. 'primary' could then pick it up and initiate communication. In the pool.yaml config you must set `inter_node_communication_enabled: true`. Also Azure Batch Shipyard populates ip addresses of the other compute nodes to the `AZ_BATCH_HOST_LIST` env variable. – rasmusgude Feb 19 '18 at 15:08
Thanks for your time and explaining the options! – jimjkelly Feb 19 '18 at 16:48

score 0 · Answer 2 · answered Feb 21 '18 at 21:24

Here is a workaround to allow this to work. Note that as a workaround, it is not officially supported and may break at any time.

Create a Docker image that has Docker itself (more specifically only the client portion is needed) installed with Docker compose.
For the tasks specification in jobs.yaml:
- docker_image is the image you created above
- Either use resource_files or input_data to ingress your compose file for the task
- additional_docker_run_options should have an item:
  - -v /var/run/docker.sock:/var/run/docker.sock
- command would be your docker-compose up command with whatever arguments you require
Ensure that your config.yaml specifies all of the Docker images required by your Docker compose file and the Docker image that contains Docker/docker-compose itself.

Running Docker Compose on Azure Batch Shipyard

2 Answers2