Stack Overflow
Stack Overflow

Questions tagged [azure-batch]

The azure-batch tag should be used to ask questions regarding all aspects of the Azure Batch service including job scheduling, compute node provisioning, and pool orchestration. Extended ecosystem questions are also welcome that are related to Azure Batch accelerator offerings, Azure Portal Batch blades, and Batch Explorer.

Azure Batch is a job scheduling and compute management orchestration solution for the Microsoft Azure cloud. The service is available in all Azure regions and can deploy with nearly all Azure VM sizes available, including those with specialized hardware such as GPUs and Infiniband/RDMA interfaces.

Azure Batch is a managed job scheduling solution where the underlying compute resources are allocated as pools. Pools may be comprised of Linux or Windows VMs and any number of dedicated and low priority nodes and can be automatically scaled through autoscale formulas. An Azure Batch Job can be thought of as a queue for tasks. Each Azure Batch Job can be assigned to a pool. Any number of tasks can be assigned to jobs, including multi-instance tasks (i.e., multi-node) which support MPI jobs. Azure Batch also supports containerized execution.

For a full introductory treatment of Azure Batch, see the Azure Batch Technical Overview. The service team also provides a GitHub repository for issue tracking, feature requests, and change logs.

416 questions
0
votes
1 answer

Azure Batch is deleting files from File Share

I have created a single Pool (size Standard_D32_v3) with a single Job. I have set the pool property max_tasks_per_node=32. I then have a list which contains 27000 objects. Since I cannot add more than 100 tasks at a time to a Job, I "Chunk" my list…
pookie
  • 3,796
  • 6
  • 49
  • 105
0
votes
1 answer

Docker Engine fails on Azure Batch node

Scenario I create a pool with multiple nodes (base image is Ubuntu Server 16.04), and provide the following start up command: /bin/bash -c 'set -o pipefail; export DEBIAN_FRONTEND=noninteractive ; sudo -E apt update ; sudo -E apt upgrade -y ; sudo…
M.S.
  • 73
  • 6
0
votes
1 answer

How do I get the path from $AZ_BATCH_NODE_SHARED_DIR?

I am using Azure Batch with Python and I would like to create a directory within the shared space from a batch task. According to the docs: Shared: This directory provides read/write access to all tasks that run on a node. Any task that runs on…
pookie
  • 3,796
  • 6
  • 49
  • 105
0
votes
1 answer

Can a batch task read files on a file share?

I have a file share with (you guessed it) a lot of files. I want to create a batch job which mounts this file share and the reads in each of the files and processes each one in parallel (each as a batch task). Is this possible to do with python and…
pookie
  • 3,796
  • 6
  • 49
  • 105
0
votes
1 answer

Azure Batch: Does StartTask run when the node is already joined to a pool

I have setup a azure batch which is a User Subscription type. Pool is already setup with 3 nodes which are in idle state. from my c# code I get the pool reference and run StartTask and do CommitAsync. Does this caused the StartTask to run or will…
kumar
  • 8,207
  • 20
  • 85
  • 176
0
votes
1 answer

Azure Batch Job: Is there a code example for batch jobs created by using User Subscription

All the examples I have seen so far user "Batch services" is there any code example for "User Subscription"? https://github.com/Azure/azure-batch-samples/tree/master/CSharp/ArticleProjects
kumar
  • 8,207
  • 20
  • 85
  • 176
0
votes
1 answer

Azure Batch : when we Disable a node do we get charged for the VM

In Azure Batch job there is an option to "Disable". When we do that do we still incur cost?
kumar
  • 8,207
  • 20
  • 85
  • 176
0
votes
1 answer

Azure Batch Job: Creating Pool for User Subscription type using C#

I would like to create Azure Batch job pool using c# for "User Subscription" I do not see an example for this where I can specify which Vnet to setup the node. I not not see any option in CreatePool to set the VNet pool =…
kumar
  • 8,207
  • 20
  • 85
  • 176
0
votes
1 answer

Azure Batch Account quota

I want to increase the quota of creating a batch account in same region. For eg. I have the limit of creating 3 batch accounts in Central US region. However I want to create 2 more batch accounts in same region. Is there any extra cost associated…
Akki
  • 161
  • 1
  • 2
  • 10
0
votes
2 answers

Job deletion and recreation in Azure Batch raises BatchErrorException

I'm writing a task manager for Azure Batch in Python. When I run the manager, and add a Job to the specified Azure Batch account, I do: check if the specified job id already exists if yes, delete the job create the job Unfortunately I fail between…
M.S.
  • 73
  • 6
0
votes
1 answer

Azure Batch Service Task Exit Code error

I downloaded the .NET Tutorial sample to get started with Azure Batch. My objective is to automate(run every 2 minutes) a web api call and store the data that I get back into a data storage container. The problem is that the Task that I am…
vpandher
  • 31
  • 2
  • 5
0
votes
0 answers

Custom .NET activity to download and extract files by using Azure Batch

I am trying to write a custom .NET activity, which will be run from Azure Data Factory. It will do two tasks, one after the other: it will download grib2 files from an FTP server daily (grib2 is a custom compression for meteorological data) it will…
FeodorG
  • 178
  • 2
  • 10
0
votes
1 answer

Retrying Tasks in Azure Batch

How does Batch know whether to retry some tasks or not? Does it solely depend on the exit code? If yes, how can I control the exit code? It seems from some logs that a task in my job got retried but I can't find the output files or information…
J.B
  • 137
  • 2
  • 16
0
votes
1 answer

Checking later the state of nodes during a past job execution

Is there a way to check in Azure Batch whether a node went to the unusable state while running a specific job on a specific pool? The context is that, when running a job and checking the pool on which it was running at that time, there were some…
J.B
  • 137
  • 2
  • 16
0
votes
1 answer

Setting up Task Constraints

How can I go about setting the max task retry count and the max task wall clock time for a task in Azure Batch? I tried the following: task.Constraints.MaxTaskRetryCount = 3; task.Constraints.MaxWallClockTime = TimeSpan.FromMinutes(10); But this…
J.B
  • 137
  • 2
  • 16
1 2 3
…
27 28