Questions tagged [azure-batch]

The azure-batch tag should be used to ask questions regarding all aspects of the Azure Batch service including job scheduling, compute node provisioning, and pool orchestration. Extended ecosystem questions are also welcome that are related to Azure Batch accelerator offerings, Azure Portal Batch blades, and Batch Explorer.

Azure Batch is a job scheduling and compute management orchestration solution for the Microsoft Azure cloud. The service is available in all Azure regions and can deploy with nearly all Azure VM sizes available, including those with specialized hardware such as GPUs and Infiniband/RDMA interfaces.

Azure Batch is a managed job scheduling solution where the underlying compute resources are allocated as pools. Pools may be comprised of Linux or Windows VMs and any number of dedicated and low priority nodes and can be automatically scaled through autoscale formulas. An Azure Batch Job can be thought of as a queue for tasks. Each Azure Batch Job can be assigned to a pool. Any number of tasks can be assigned to jobs, including multi-instance tasks (i.e., multi-node) which support MPI jobs. Azure Batch also supports containerized execution.

For a full introductory treatment of Azure Batch, see the Azure Batch Technical Overview. The service team also provides a GitHub repository for issue tracking, feature requests, and change logs.

416 questions
1
vote
1 answer

How to load python libraries in Azure Data Factory Custom Activity?

I understand you can use a batch custom activity with custom VM image with preloaded python libraries to run your python programs through ADF. Can we also use a plain linux VM image and load the python libraries as part of the execution of the…
Gadam
  • 2,674
  • 8
  • 37
  • 56
1
vote
1 answer

Azure Batch Preempted state

I have a TVM/pool running under Azure batch and suddenly it went into the Preempted state. Now the problem is, it is not taking any requests now. I have also setup Scale formula wherein it gives me a VM whenever I have more then 0 job pending to be…
Gagan
  • 1,775
  • 5
  • 31
  • 59
1
vote
1 answer

Azure Batch :Elevating the user privileges during Pool Creation using Azure CLI

I need to mount the azure file storage to Linux-Pools when they are being spun-up.I am following the instructions given here to achieve that: mounting Azure-File Storage to Batch Specically in my Azure CLI script under the Pools start commands I am…
user8229596
1
vote
1 answer

Unable to run tasks on Azure Batch:Nodes go into unusable state after starting-up

I am trying to parallelize a Python App using Azure Batch.The workflow that I have followed in the Python client-side script is: 1) Upload Local files to Azure Blob Container using blobxfer utility (input-container) 2) Start Batch service to Process…
user8229596
1
vote
1 answer

Custom Image under AzureBatch ImageReference class not working

I have a custom VHD file with me. I am able to create Pool with my custom image through portal. But i want to try the same with .Net SDK. But it is throwing error "Operation returned an invalid status code 'Forbidden". I am referring this link …
user1037747
  • 1,307
  • 2
  • 18
  • 38
1
vote
0 answers

Most suitable Azure service make thousands of outbound API calls

I am currently working on a project and trying to find out what is the most suitable Azure option to make daily API calls to Dark Sky(https://darksky.net/dev) In short we are looking to bring weather data inside our data lake storage(Azure Data Lake…
1
vote
0 answers

Apache Beam Runner for Azure Batch

Azure Batch on its own doesn't have MapReduce APIs like Spark. I am wondering if any of the Runners provided by Apache Beam can be used to start a MapReduce job on the pool of VMs created under Azure Batch. Has anyone got Apache Beam APIs to work…
DilTeam
  • 2,551
  • 9
  • 42
  • 69
1
vote
1 answer

Azure batch working directory issue

I am newbie to Azure batch as well as docker. The problem i am facing is i have created an image based on another custom image in which some files and folders are created at the root level/directory of the container and every thing works fine but…
1
vote
2 answers

Unable to host docker image from azure registry to azure batch

I am new to docker as well as azure batch. The problem i am having currently is i have 2 dotnet console applications one of them runs locally (which creates the pool, job and task on azure batch programmatically) and for second one i have created a…
1
vote
1 answer

How to set up Azure Batch pool container configuration via ARM template?

We are using Azure Batch, and we need to use Windows Docker containers on the VMs. This is how it is done via Portal: And this is how it is done via C# API: private static VirtualMachineConfiguration ConfigureVM() { var imageNames = new…
psfinaki
  • 1,814
  • 15
  • 29
1
vote
1 answer

Add Tasks to a running Azure batch job and manually control termination

We have an Azure-batch job that uses some quite large files which we are uploading to Azure Blob storage asynchronously so that we don't have to wait for all files to upload before starting our batch job made up of a collection of Tasks that will…
1
vote
1 answer

Azure batch - run multiple commands from command line on linux VM

At the beginning, I created a pool with the following configuration: "Publisher": "microsoft-ads" "Offer": "linux-data-science-vm" "Sku": "linuxdsvm" Then I want "start.sh" file to be copied and run on VM. So, I need to change permissions, to…
Andrei Barbolin
  • 395
  • 1
  • 3
  • 13
1
vote
1 answer

How to see if azure batch job exists from azure cli?

Currently I do the following to see if a job exists: id=$(az batch job show \ --job-id $1 \ --query id \ --output tsv \ --account-key $AZ_BATCH_KEY \ --account-name $AZ_BATCH_ACCOUNT \ --account-endpoint…
Dan McGrath
  • 145
  • 10
1
vote
1 answer

How to add application packages to Azure Batch task from Azure CLI?

I am trying to write a bash command line script that will create an azure batch task with an application package. The package is called "testpackage" and exists and is activated on the batch account. However, every time I create this task, I get the…
Dan McGrath
  • 145
  • 10
1
vote
2 answers

Running Docker Compose on Azure Batch Shipyard

I was wondering if anyone had any idea how to run compose on Batch Shipyard, or short of directly using compose, allowing multiple containers to work in concert on the same node in a job. Is this possible? To clarify - I have several containers…
jimjkelly
  • 1,631
  • 14
  • 15