Stack Overflow
Stack Overflow

Questions tagged [azure-batch]

The azure-batch tag should be used to ask questions regarding all aspects of the Azure Batch service including job scheduling, compute node provisioning, and pool orchestration. Extended ecosystem questions are also welcome that are related to Azure Batch accelerator offerings, Azure Portal Batch blades, and Batch Explorer.

Azure Batch is a job scheduling and compute management orchestration solution for the Microsoft Azure cloud. The service is available in all Azure regions and can deploy with nearly all Azure VM sizes available, including those with specialized hardware such as GPUs and Infiniband/RDMA interfaces.

Azure Batch is a managed job scheduling solution where the underlying compute resources are allocated as pools. Pools may be comprised of Linux or Windows VMs and any number of dedicated and low priority nodes and can be automatically scaled through autoscale formulas. An Azure Batch Job can be thought of as a queue for tasks. Each Azure Batch Job can be assigned to a pool. Any number of tasks can be assigned to jobs, including multi-instance tasks (i.e., multi-node) which support MPI jobs. Azure Batch also supports containerized execution.

For a full introductory treatment of Azure Batch, see the Azure Batch Technical Overview. The service team also provides a GitHub repository for issue tracking, feature requests, and change logs.

416 questions
1
vote
1 answer

best practice for datasets conversions for usage in AML

What are the best practices for large datasets conversion? In many of the cases I deal with there is always a first step where the input dataset is converted to a format that is consumable by the training (I deal with thousands of images). The…
user9427997
  • 43
  • 1
  • 4
1
vote
1 answer

How I can call pre-installed application in Azure Batch task?

I installed sqlcmd utility into Azure Batch Node (this is regular windows VM). So, I have bcp utility on this VM. How I can specify path to bcp.exe in Azure Batch job? using (BatchClient batchClient = BatchClient.Open(cred)) { string jobId =…
Artem Beziazychnyi
  • 550
  • 6
  • 15
1
vote
1 answer

Saving output while tasking is running in Azure Batch (using python SDK)

I want to save the output of a task while it is running. I was able to save the output only after the task completion. I found in the C# azure sdk documentation that it has a SaveTrackedAsync function that enables to track a file and save it to a…
Gustavo
  • 58
  • 5
1
vote
1 answer

Can I configure Azure Batch to use Nvidia VM images?

I'm trying to configure an Azure Batch pool to use "PyTorch from NVIDIA" VM images. I'm provisioning the pool using the Azure Python SDK and I'm getting the following error: Reason: The specified imageReference with publisher nvidia offer…
ifeins
  • 869
  • 1
  • 9
  • 20
1
vote
1 answer

Generating ZIP files in azure blob storage

What is the best method to zip large files present in AZ blob storage and download them to the user in an archive file (zip/rar) does using azure batch can help ? currently we implement this functions in a traditionally way , we read stream generate…
Houssem
  • 1,042
  • 7
  • 16
1
vote
2 answers

How to authenticate with Key Vault from Azure Batch

I've been following this guide to use a certificate to authenticate with key vault from azure batch. Every certificate I generate causes errors on import into azure batch, some examples are listed below: code : InvalidPropertyValue message : The…
Neil P
  • 2,920
  • 5
  • 33
  • 64
1
vote
1 answer

How to list all tasks in an Azure Batch account using the .NET API?

In my Azure Batch account, I run many jobs simultaneously. How can I get all tasks across all jobs?
Zach Esposito
  • 707
  • 9
  • 17
1
vote
2 answers

Adding Tags To Azure Batch Resources

Is it possible to tag the resources used from azure batch? I use the .NET SDK to create a batch pool, job and tasks. I'm wondering if I can tag the pool VM's that get created by the azure batch client. I looked through the properties and…
MadSkeletor
  • 161
  • 1
  • 14
1
vote
2 answers

How to install .net core on azure batch node?

I am trying to install .net core 2.2 on azure batch node using start up task ? The command that I am trying to use in startup task is below: @powershell -NoProfile -ExecutionPolicy unrestricted -Command "[Net.ServicePointManager]::SecurityProtocol =…
Mike143
  • 163
  • 3
  • 14
1
vote
4 answers

How do I Unzip and Execute a Batch Service job as part of Azure Data Factory

Azure Data Factory can execute custom activities as Batch Service Jobs. These jobs can run from an .exe (and associated dependencies) in a storage account which are copied across prior to execution. There is a limitation on the files in the storage…
Cargowire
  • 1,488
  • 10
  • 21
1
vote
1 answer

Unziping input files before running tasks

I want to be able to unzip an input file before running tasks to reduce data transfer in my batching process. To do this I added the following command line to my JobPreparationTask: CommandLine = "/bin/bash -c 'DEBIAN_FRONTEND=noninteractive…
SandiaDeDia
  • 291
  • 2
  • 9
1
vote
1 answer

R doAzureParallel and chunkSize. Is there a new R session per loop iteration?

Intend to use Azure Batch to run R in parallel with the do AzureParallel package. I intend leave the default chunkSize = 1. Does it start a new R session for each iteration of the foreach() loop? I do want it to start a new R session for each…
power
  • 1,680
  • 3
  • 18
  • 30
1
vote
1 answer

Detecting the end of an Azure Batch job

In my application I create an Azure batch job. It's a Node app and I use an azure-batch Node client, but I could also be using REST, I don't think it matters. I can't switch to a C# client, however. I expect the job to be completed in a few seconds…
Avius
  • 5,504
  • 5
  • 20
  • 42
1
vote
1 answer

Parallel Tasks in Data Factory Custom Activity (ADF V2)

I am running Custom code activity in ADF v2 using Batch Service. Whenever this runs it only create one CloudTask within my Batch Job although I have more than two dozen parallel.Invoke methods running. Is there a way I can create multiple Tasks from…
InTheWorldOfCodingApplications
  • 2,526
  • 6
  • 46
  • 87
1
vote
1 answer

Container task settings for azure batch task Using CLI

I am trying a task in azure batch, I want to add Container settings for the batch task using CLI. I don't see any information on how to add the container settings using CLI. can anyone help me?
Raga
  • 11
  • 2
1 2 3
…
27 28