Questions tagged [azure-batch]

The azure-batch tag should be used to ask questions regarding all aspects of the Azure Batch service including job scheduling, compute node provisioning, and pool orchestration. Extended ecosystem questions are also welcome that are related to Azure Batch accelerator offerings, Azure Portal Batch blades, and Batch Explorer.

Azure Batch is a job scheduling and compute management orchestration solution for the Microsoft Azure cloud. The service is available in all Azure regions and can deploy with nearly all Azure VM sizes available, including those with specialized hardware such as GPUs and Infiniband/RDMA interfaces.

Azure Batch is a managed job scheduling solution where the underlying compute resources are allocated as pools. Pools may be comprised of Linux or Windows VMs and any number of dedicated and low priority nodes and can be automatically scaled through autoscale formulas. An Azure Batch Job can be thought of as a queue for tasks. Each Azure Batch Job can be assigned to a pool. Any number of tasks can be assigned to jobs, including multi-instance tasks (i.e., multi-node) which support MPI jobs. Azure Batch also supports containerized execution.

For a full introductory treatment of Azure Batch, see the Azure Batch Technical Overview. The service team also provides a GitHub repository for issue tracking, feature requests, and change logs.

416 questions
2
votes
1 answer

Fetching Azure Batch account access keys in ARM template

In my ARM template, I want to fetch primary key for batch account specified by batch_accountName parameter. Expression that should fetch keys for batch account is: "[listKeys(resourceId('Microsoft.Batch/batchAccounts',…
2
votes
2 answers

How to use Azure Batch in an event based design and terminate/cleanup finished jobs

Using Azure Batch, my project adds jobs to a pool using an event based design with functions and queues. When the job is finished, it is still "active", even though all tasks are completed. A (single using an app service plan) function is triggered…
JeroenW
  • 753
  • 6
  • 16
2
votes
1 answer

How to run multiple command line in Azure Pool start task

I'm trying to run this command line in Azure Batch node start up task sudo wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie"…
Quan Hoang
  • 45
  • 5
2
votes
0 answers

Azure Data Factory v2 and data processing in custom activity

I am migrating (extract-load) a large dataset to a LOB service, and would like to use Azure Data Factory v2 (ADF v2). This would be the cloud version of the same kind of orchestration typically implemented in SSIS. My source database and dataset, as…
krembanan
  • 1,408
  • 12
  • 28
2
votes
2 answers

Azure batch Application package not getting copied to Working Directory of Task

I have created Azure Batch pool with Linux Machine and specified Application Package for the Pool. My command line is command='python $AZ_BATCH_APP_PACKAGE_scriptv1_1/tasks/XXX/get_XXXXX_data.py', python3: can't open file…
NSS
  • 1,835
  • 2
  • 29
  • 66
2
votes
1 answer

AD Service Principal for Azure Container Registry from Azure Batch container configuration?

I am trying to use Docker images from an Azure Container Registry as tasks in Azure Batch. In the Docker CLI, I can authenticate to the ACR using an Active Directory Service Principal's credentials, with the application ID as the username and the…
2
votes
1 answer

Azure Batch tasks stuck in Running state

I have several tasks on Azure Batch which are stuck in Running state although the node server does not know anything about it (not running there, no folders found). Any task manipulation in GUI (Terminate, Delete, Show files on node) end with There…
Marki555
  • 6,434
  • 3
  • 37
  • 59
2
votes
1 answer

Perform a task on Azure batch

I am new to Azure batch. I have to perform a task on nodes in the pool. The approach I am using is that I have the code that I want to run on the node. I am making a zip of the jar of the .class file and uploading to my Azure storage account and…
Tarun
  • 271
  • 1
  • 6
  • 18
2
votes
1 answer

How to increase the "Max tasks per node" in Azure batch?

I have an Azure batch account and I'd like to increase the "Max tasks per node" property on my pool. How do I do this?
Neil P
  • 2,920
  • 5
  • 33
  • 64
2
votes
1 answer

How to install Python packages on Azure Batch Node

I am following the MS Azure Batch tutorial for Python: https://learn.microsoft.com/en-us/azure/batch/batch-python-tutorial Now, I want to use some non-standard Python packages, such as SciPy or Colorama... whatever. for example: import scipy import…
pookie
  • 3,796
  • 6
  • 49
  • 105
2
votes
0 answers

Azure Batch : How to set static IP for nodes

How do we set static IP for the nodes in Azure Batch jobs. Out batch job needs to access our sql server during the batch job. Our SQL server is on a Classic VM with ACL on. Due to this we need to add IP to the SQL server ACL so that the batch job…
kumar
  • 8,207
  • 20
  • 85
  • 176
2
votes
1 answer

Graceful custom activity timeout in data factory

I'm looking to impose a timeout on custom activities in data factory via the policy.timeout property in the activity json. However I haven't seen any documentation to suggest how the timeout operates upon Azure batch? I assume that the batch task is…
Alex KeySmith
  • 16,657
  • 11
  • 74
  • 152
2
votes
1 answer

How can I programmatically get the number of cores in an Azure Batch Node?

We are using the Azure Batch service to run some computation engine code, and we specify the size of of the VM when we create the pool: var pool = batchClient.PoolOperations.CreatePool(poolId: $"{PoolIdPrefix}-{poolGuid}", virtualMachineSize:…
B. Witter
  • 564
  • 6
  • 19
2
votes
1 answer

Application Packages with VM configuration

I'm trying to use application packages in the way they're described in this https://learn.microsoft.com/en-us/azure/batch/batch-application-packages But I keep getting an error saying application path not found. Any ideas what could be wrong?…
J.B
  • 137
  • 2
  • 16
2
votes
2 answers

Local VM storage in Azure Batch task

For my task I'm using locally persisted data. Until now I've successfully used Path.GetTempPath() to get temp folder and persist data there and perform some computations. The temp folder is on the system drive which is not big enough (around 30GB).…
mcs_dodo
  • 738
  • 1
  • 10
  • 17