Stack Overflow
Stack Overflow

Questions tagged [azure-batch]

The azure-batch tag should be used to ask questions regarding all aspects of the Azure Batch service including job scheduling, compute node provisioning, and pool orchestration. Extended ecosystem questions are also welcome that are related to Azure Batch accelerator offerings, Azure Portal Batch blades, and Batch Explorer.

Azure Batch is a job scheduling and compute management orchestration solution for the Microsoft Azure cloud. The service is available in all Azure regions and can deploy with nearly all Azure VM sizes available, including those with specialized hardware such as GPUs and Infiniband/RDMA interfaces.

Azure Batch is a managed job scheduling solution where the underlying compute resources are allocated as pools. Pools may be comprised of Linux or Windows VMs and any number of dedicated and low priority nodes and can be automatically scaled through autoscale formulas. An Azure Batch Job can be thought of as a queue for tasks. Each Azure Batch Job can be assigned to a pool. Any number of tasks can be assigned to jobs, including multi-instance tasks (i.e., multi-node) which support MPI jobs. Azure Batch also supports containerized execution.

For a full introductory treatment of Azure Batch, see the Azure Batch Technical Overview. The service team also provides a GitHub repository for issue tracking, feature requests, and change logs.

416 questions
0
votes
1 answer

When using Azure Batch Processing, what is the best way to create and use a configuration file which can change per instance?

So I'm new to Azure and currently working on a project in which I will be using azure batch processing to run an application in several instances with different configurations. I was wondering what is the best practice for doing this, with reference…
gn12345
  • 335
  • 1
  • 3
  • 11
0
votes
1 answer

Data Factory job failing to submit to Azure Batch?

I'm trying to submit a Data Factory pipeline to Azure Batch compute, with a linked service that I have previously been using and works fine. However, the pipeline is failing with the following message: Azure Batch entity not found. Code:…
Tayyab Anwar
  • 319
  • 1
  • 10
0
votes
2 answers

Azure Batch - Create Nodes from VM Image

I have some Selenium test code that I need to run in parallel. In order for Selenium to run effectively, certain configurations have to be done on the machine (I.E. zone settings, Chrome and Firefox installs, etc.) and these settings are hard (if…
user4794170
  • 203
  • 1
  • 4
  • 8
0
votes
1 answer

Azure Batch Apps install python packages on startup

We are using Azure Batch Apps which will create multiple VMs which can be used to run our tasks parallelly. We are using python for data fetching tasks. We have mentioned in the batch apps to install anaconda on the VMs when they start up. Anaconda…
Selva Saravana Er
  • 199
  • 1
  • 1
  • 6
0
votes
2 answers

moving files from a batch job to blob

I am executing an azure-batch job, which creates a zip file as its output. The batch is being executed by an orchestrator with the responsibility of moving the output files to blob. I have this working, but it feels clunky - i.e. I download the…
NDJ
  • 5,189
  • 1
  • 18
  • 27
0
votes
1 answer

Azure Batch services scheduling options

I am having hard time in deciding the right Job Scheduling for Azure Batch Services. Does anyone have a thought to share on what could be used ? I came up with only 2 options: JobSchedule Class…
JackMith
  • 45
  • 5
0
votes
1 answer

How to round in Azure batch autoscale formula

When I commit my formula, Azure complains about calculation errors. I narrowed it down to having a number with decimal places. Here’s my formula: pivot = (0.15*($CurrentDedicated/20)+0.84); target =…
Nomenator
  • 1,021
  • 1
  • 13
  • 28
0
votes
1 answer

create custom directory in Azure batch

I'm uploading a file from blob storage to an Azure Task which gets uploaded to the task VM's wd directory. I can access the file fine from the executing task. I upload the file with: var filename = "myFile.txt"; ResourceFile myResource = new…
NDJ
  • 5,189
  • 1
  • 18
  • 27
0
votes
1 answer

Download Files from Azure Batch

I would like to download the files after the task is complete. How can I do it efficiently? Unfortunately, task.EnvironmentSettings is null and the following code snipet does not work (see the Microsoft instructions). IList…
luksmir
  • 3,104
  • 5
  • 22
  • 38
0
votes
1 answer

Changing Azure VM OS Family dynamically

I am designing an application to connect to Azure Batch and run some compute intense jobs on demand for our users. When you create the pool you can specify the OS family and Version of VMs but what I am wondering is can you change this after the…
NZJames
  • 4,963
  • 15
  • 50
  • 100
-1
votes
1 answer

Which Azure service for batch processing?

Once a day I want to download multiple CSV files (could be several hundred of them), parse them, do some transforming and write them to another file or database. I want to have an overview which processing failed and which succeeded and prefereably…
stefan.at.kotlin
  • 15,347
  • 38
  • 147
  • 270
-1
votes
2 answers

How to check if the batch pool nodes are idle in data factory

I am using a Python code that creates the batch pool using the Azure SDK for Python. I have deployed this code in function apps and I am using it in data factory to create the batch pool. I want to check if the nodes have started or if they are in…
Sindu_
  • 1,347
  • 8
  • 27
  • 67
-1
votes
2 answers

How to specify the task's user identity "Pool default user (Admin)" in the azure batch account?

I'm trying to create several batch tasks from the Powershell script and looking for how specifying User Identity. Dotnet core APIs support this specifying, but nothing for Azure-CLI. When I run a script below I get an error from my PowerShell…
Joe Black
  • 23
  • 1
  • 1
  • 5
-1
votes
1 answer

Update Azure Batch application from build pipeline

How can one update an Azure Batch application with a task in a build pipeline on AzureDevOps?
participant
  • 2,923
  • 2
  • 23
  • 40
-1
votes
1 answer

How to upload to blob container from Job Release Task in Azure Batch .net

I need to upload files to Blob Container in Azure batch service. I know I can do that in CloudTask using OutpuFiles but I want to do that in JobRelease Task. I cant figure out how can I do that. I am using C# SDK for azure batch.
Rajat Sharma
  • 47
  • 1
  • 15
1 2 3
…
27
28