6

I have a Docker application that works fine in my laptop on Windows using compose and starting multiple instances of a container as a Dask cluster.

The name of the service is "worker" and I start two container instances like so:

docker compose up --scale worker=2

I deployed the image on Azure and when I run docker compose (using the same command I used in Windows) only one container is started.

How to deploy a cluster of containers in Azure? Can I use docker compose or I need to have a different approach, such as deploying with templates or Kubernetes?

This is the docker-compose.yml file:

version: "3.0"
services:

  web:
    image: sofacr.azurecr.io/pablo:job2_v1
    volumes:
      - daskvol:/code/defaults_prediction      
    ports:
      - "5000:5000"
    environment:
      - SCHEDULER_ADDRESS=scheduler
      - SCHEDULER_PORT=8786
    working_dir: /code
    entrypoint:
      - /opt/conda/bin/waitress-serve
    command:
      - --port=5000
      - defaults_prediction:app

  scheduler:
    image: sofacr.azurecr.io/pablo:job2_v1
    ports:
      - "8787:8787"
    entrypoint:
      - /opt/conda/bin/dask-scheduler

  worker:
    image: sofacr.azurecr.io/pablo:job2_v1
    depends_on:
      - scheduler
    environment:
      - PYTHONPATH=/code
      - SCHEDULER_ADDRESS=scheduler
      - SCHEDULER_PORT=8786
    volumes:
      - daskvol:/code/defaults_prediction
      - daskdatavol:/data
      - daskmodelvol:/model
    entrypoint:
      - /opt/conda/bin/dask-worker
    command:
      - scheduler:8786
      
volumes:
  daskvol:
    driver: azure_file
    driver_opts:
     share_name: daskvol-0003
     storage_account_name: sofstoraccount
  daskdatavol:
    driver: azure_file
    driver_opts:
     share_name: daskdatavol-0003
     storage_account_name: sofstoraccount
  daskmodelvol:
    driver: azure_file
    driver_opts:
     share_name: daskmodelvol-0003
     storage_account_name: sofstoraccount
ps0604
  • 1,227
  • 23
  • 133
  • 330
  • What specific azure service are you using? Does docker report any errors? – mdurant Apr 22 '21 at 18:35
  • I'm using the container registry and instances, deploy with `docker compose up --scale worker=2` connected to the Azure context from Windows. There are no errors, a single container is deployed and it works fine – ps0604 Apr 22 '21 at 19:16
  • Azure Container instances with the docker CLI, as documented at https://learn.microsoft.com/en-ca/azure/container-instances/quickstart-docker-cli ? – mdurant Apr 22 '21 at 20:18
  • [This is the tutorial I followed](https://learn.microsoft.com/en-us/azure/container-instances/tutorial-docker-compose) – ps0604 Apr 22 '21 at 20:26
  • Any update on this question? Does it solve your problem? – Charles Xu Apr 26 '21 at 01:05

3 Answers3

3

What you need here is Azure Kubernetes Service or Azure Webapps for Containers. Both will take care of taking Docker images from ACR and distributing then across a fleet of machines.

Here is a decision tree to choose your compute service

Container Instances - small, fast, serverless container hosting service - usually nice for small container deployments, I tend to use it to spawn adhoc background jobs

AKS - large scale container deployment, big part here is the multi-container orchestration platform . Have a look at this example

Sajeetharan
  • 216,225
  • 63
  • 350
  • 396
1

I am also new to docker . But as far I read to orchestrate the containers or to scaleup the container generally use the docker swarm or kubernetes. In Azure kuberenetes cluster is AKS.

docker compose up --scale worker=2

I have come across this issue of scaling container in this link https://github.com/docker/compose/issues/3722

How to simply scale a docker-compose service and pass the index and count to each? hope this might help.

lavanya
  • 106
  • 3
1

Now ACI supports deploying from the docker-compose.yaml file, but it doesn't support scaling up to multiple replicas for the same container. Because ACI does not support port mapping and one port can only expose once and for one container. So you only can create multiple containers in a container group through the docker-compose. yaml file and each container have one replica.

If you want to have multiple replicas of one container, then I recommend you use the AKS, it's more suitable for your purpose.

Charles Xu
  • 29,862
  • 2
  • 22
  • 39