Stack Overflow
Stack Overflow

Questions tagged [azure-batch]

The azure-batch tag should be used to ask questions regarding all aspects of the Azure Batch service including job scheduling, compute node provisioning, and pool orchestration. Extended ecosystem questions are also welcome that are related to Azure Batch accelerator offerings, Azure Portal Batch blades, and Batch Explorer.

Azure Batch is a job scheduling and compute management orchestration solution for the Microsoft Azure cloud. The service is available in all Azure regions and can deploy with nearly all Azure VM sizes available, including those with specialized hardware such as GPUs and Infiniband/RDMA interfaces.

Azure Batch is a managed job scheduling solution where the underlying compute resources are allocated as pools. Pools may be comprised of Linux or Windows VMs and any number of dedicated and low priority nodes and can be automatically scaled through autoscale formulas. An Azure Batch Job can be thought of as a queue for tasks. Each Azure Batch Job can be assigned to a pool. Any number of tasks can be assigned to jobs, including multi-instance tasks (i.e., multi-node) which support MPI jobs. Azure Batch also supports containerized execution.

For a full introductory treatment of Azure Batch, see the Azure Batch Technical Overview. The service team also provides a GitHub repository for issue tracking, feature requests, and change logs.

416 questions
1
vote
1 answer

Azure Batch microtask parallel processing (Modify Task Queue)

I am trying to parallelise micro-tasks that will be fired off on one of the VMs and should get parallelised on all the VMs. How can I modify the Azure Batch queue. Is there any way to add a task to the queue through the API?
Sagar Thapar
  • 11
  • 1
1
vote
1 answer

Persisting Azure Batch task output to Azure storage

Say I have a bunch of similar tasks run in parallel in Azure Batch pool of VMs. These tasks connect to SQL database and extract data for individual table using sqlcmd. Then table output is compressed by piping it to 7zip.exe. So, command line is…
YuGagarin
  • 341
  • 7
  • 20
1
vote
2 answers

Azure Batch Linux vm running script

I'm trying to use Azure batch services by creating linux vm. I have some computations for which I'm using python.To install some python packages, I have placed the install commands in script file and try to run that in startup. pool.StartTask =…
Selva
  • 976
  • 1
  • 10
  • 23
1
vote
1 answer

Azure Batch support for Data Lake Store Linked Service

I am using a data factory pipeline with a custom activity (configured to run on Azure Batch) that has a data lake store input dataset and output dataset. The data lake store linked service is using service to service auth (service principal) and is…
Sebastian
  • 13
  • 2
1
vote
1 answer

Do Azure Batch jobs need a watcher process?

We have a very long running operation (potentially days) that we would like to have triggered from a BLOB file written to a Azure Storage. This job could be started once year, never, or many times over a few days. Azure Batch jobs look exactly like…
Lane Goolsby
  • 594
  • 1
  • 8
  • 26
1
vote
1 answer

Azure Batch Pool Start up task to download resource file from Blob FileShare

I'm new to Azure, and I'm trying to build a simple Azure batch case here. I'm stacking at the Batch Pool Start up task... I created a batch account and a storage account in East US, and then I created a general -- Fileshare in the storage account…
kris_zhui
  • 13
  • 1
  • 5
1
vote
1 answer

How do I add existing VM in Azure Batch pool?

We have one Windows 2012 based VM in Azure and want to add this VM into Azure Batch service's pool. Reason behind this is that we have installed one licensed application on this VM. This application provides installation wizard and I am not sure if…
Shekhar
  • 11,438
  • 36
  • 130
  • 186
1
vote
1 answer

On-Prem SQL connection throwing SqlException in Datafactory custom activity

I have added code for Azure datafactory custom activity in Azure batch service and pointed the datafactory pipeline to the bacth service. When I execute the code in local environment, it works fine. But when I upload it to run in azure batch…
Maverik
  • 411
  • 4
  • 22
1
vote
2 answers

Use Microsoft Azure as a computing cluster

My lab just got a sponsorship from Microsoft Azure and I'm exploring how to utilize it. I'm new to industrial level cloud service and pretty confused about tons of terminologies and concepts. In short, here is my scenario: I want to experiment the…
Higerra
  • 29
  • 3
1
vote
1 answer

How to list Azure Batch Pools from PowerShell using the account key?

Using the Azure Batch PowerShell cmdlets, I would like to list the pools within an existing Azure Batch account using the Azure Batch account primary key. The documentation hints how to do that with full login to the Azure subscription, however, I…
Joannes Vermorel
  • 8,976
  • 12
  • 64
  • 104
1
vote
2 answers

Create azure batch pool using Azure Batch Restful API, encounter exception

I am trying to create a pool using RESTful API. I know there is C# library for batch service, but in order to programmingly specify the subnet id, I have to use RESTful API to create it which I read about in this MSDN article. My Post URI follow the…
esther fang
  • 149
  • 2
  • 12
1
vote
2 answers

Run time-consuming .exe from Microsoft Azure

I have an executable that performs long calculations and I want to run those calculations on Azure. What would be the optimal service - batch or VM perhaps?
user6358539
  • 13
  • 3
1
vote
1 answer

Azure Batch Job How to split large file into smaller files

Want to use Azure Batch Job to process large files, but cannot figure out how and where will i split my large file into smaller file and how do i pass that into Task so that same file is not passed to any other instance of Task
NSS
  • 1,835
  • 2
  • 29
  • 66
1
vote
2 answers

Worker Role vs Azure Batch

We have a scenario for on prem to upload a bunch of pdf/tiffs then service on the cloud to process them. In the same time a queue is populated with metadata on processing instructions. We are trying to decide if whether the worker role or the Azure…
TOMMY WANG
  • 1,382
  • 3
  • 16
  • 39
1
vote
1 answer

Which DLLs are requied for task in Azure Batch?

My question is related to Azure Batch example. Therefore if you do not examine the example, my question will not make any sense. In this example, total seven NuGet packages are required for the TaskApplication project. However when uploading to…
Nuri Tasdemir
  • 9,720
  • 3
  • 42
  • 67
1 2 3
…
27 28