Stack Overflow
Stack Overflow

Questions tagged [azure-batch]

The azure-batch tag should be used to ask questions regarding all aspects of the Azure Batch service including job scheduling, compute node provisioning, and pool orchestration. Extended ecosystem questions are also welcome that are related to Azure Batch accelerator offerings, Azure Portal Batch blades, and Batch Explorer.

Azure Batch is a job scheduling and compute management orchestration solution for the Microsoft Azure cloud. The service is available in all Azure regions and can deploy with nearly all Azure VM sizes available, including those with specialized hardware such as GPUs and Infiniband/RDMA interfaces.

Azure Batch is a managed job scheduling solution where the underlying compute resources are allocated as pools. Pools may be comprised of Linux or Windows VMs and any number of dedicated and low priority nodes and can be automatically scaled through autoscale formulas. An Azure Batch Job can be thought of as a queue for tasks. Each Azure Batch Job can be assigned to a pool. Any number of tasks can be assigned to jobs, including multi-instance tasks (i.e., multi-node) which support MPI jobs. Azure Batch also supports containerized execution.

For a full introductory treatment of Azure Batch, see the Azure Batch Technical Overview. The service team also provides a GitHub repository for issue tracking, feature requests, and change logs.

416 questions
3
votes
1 answer

Azure Batch REST API, authorization issue

I've tried to authenticate to Azure Batch using REST API, to do so I wrote following PowerShell code $Key = 'key' $region = "region" $sharedKey = [System.Convert]::FromBase64String($Key) $date = [System.DateTime]::UtcNow.ToString("R") $stringToSign…
Adam Marczak
  • 2,257
  • 9
  • 20
3
votes
1 answer

how to hit a flask app running in azure batch node using a task?

I was able to configure the following setup, [] I am able to add a task in a job to run the testapi.py file which takes a simple string as input and send it to the flask app running inside the docker container. But, The task execution is throwing…
cerofrais
  • 1,117
  • 1
  • 12
  • 32
3
votes
1 answer

How to update stdout.txt during task execution (real time)?

I am using Azure Batch to execute a process which has different steps, the process is programmed in Python. I want to keep a log of the steps my process is in in the stdout.txt file in Azure Batch. So far, I am printing the step I am in. With this,…
Marisa
  • 1,135
  • 3
  • 20
  • 33
3
votes
1 answer

Unable to execute a powershell script using Azure Batch Account task

I am trying to execute a powershell script (that I am uploading to the storage account using the Azure Storage Client Library for .NET) from a task that runs within a Batch pool. I know that I am uploading the powershell script correctly (I can…
Barry Roberts
  • 33
  • 4
3
votes
1 answer

How to run DotNet Core applications in Azure Batch Jobs

I have followed Azure Batch ffmpeg tutorial and it works as expected. The problem that I have at this point is that my executable that i want to run as a batch is a dotnet core application and images that are available when creating a Pool do not…
tridy
  • 1,166
  • 1
  • 12
  • 21
3
votes
2 answers

Run a python script on Azure Batch

Iam trying to execute a python script on azure batch which is a linux dsvm so that the script can install python packages and then execute the python script. Below is the code i used: try: from pip import main as pipmain except ImportError: …
Matt_85
  • 41
  • 1
  • 4
3
votes
1 answer

Need help reimaging an existing Azure Node in a Azure Batch Pool with Virtual Machine Configuration

Does anyone have advice on how to reimage Linux nodes in my Azure Batch Pool account without resizing to 0 then back to N, or deleting the pool and creating it again? Or is that the recommended best practice More details: I'm having issues with…
Pranshu
  • 33
  • 3
3
votes
1 answer

Azure Batch - Task competes but does nothing

I'm trying to run a task on Azure Batch pool. The task completes with no errors, but produces no output. I created and configured Ubuntu 18.04 LTS virtual machine, put my .NET Core 2.1 SCA into /usr/local/bin/MyApp and created image from this…
270F
  • 119
  • 2
  • 12
3
votes
1 answer

How can an ADF Custom Activity access the Azure Key Vault?

I have a custom activity that runs through and ADF pipeline, with an Azure Batch for the execution. The custom activity needs to access the Azure Key Vault, but I don't want to leave any secrets/keys coded in the custom activity. How can I get the…
Anders
  • 894
  • 2
  • 10
  • 25
3
votes
2 answers

How to authenticate with Azure Analysis Services from Azure batch and data factory

I have a c sharp class library that connects to Azure Analysis Services using the AMO library. I'd like to use this as part of my data factory pipeline to refresh cube partitions. This is done through Azure batch as a custom .net activity. var…
Neil P
  • 2,920
  • 5
  • 33
  • 64
3
votes
2 answers

Upload Azure Batch Job Application Package programmatically

I have found how to upload/manage Azure Batch job Application Packages through the UI: https://learn.microsoft.com/en-us/azure/batch/batch-application-packages And how to upload and manage Resource Packages…
Lane Goolsby
  • 594
  • 1
  • 8
  • 26
3
votes
1 answer

Updating Azure Batch autoscale formula once pool exists

I can create a pool with an autoscale formula fine. The code for this is as follows. var pool = client.PoolOperations.CreatePool(poolName, vmsize, new CloudServiceConfiguration(osFamily, osVersion)); pool.TaskSchedulingPolicy = new…
NZJames
  • 4,963
  • 15
  • 50
  • 100
2
votes
1 answer

Creating an Azure Batch Service Pool using an ADF pipeline

I want to automatically create an Azure Batch Service pool when an ADF pipeline is run. I created a web activity inside the pipeline and passed the parameters under settings as follows: URL: https://management.azure.com/subscriptions/{Subscription…
Vivek
  • 336
  • 2
  • 4
  • 18
2
votes
1 answer

Periodic stdout and stderr from azure batch service

I'm running a pipeline in azure data factory, and I'm using a custom cell to run a azure batch activity. The azure batch job I run is really big, and I would like to monitor which stage I'm in that job. On a remote VM, I typically do this using the…
tandem
  • 2,040
  • 4
  • 25
  • 52
2
votes
2 answers

Azure/Python - download files quickly from storage

my Azure webapp needs to download 1000+ very small files from a blob storage directory and process them. If I list them, then download them one by one, it takes ages... Is there a fast way to do it? Like to download them all together? PS: I use the…
user1403546
  • 1,680
  • 4
  • 22
  • 43
1
2
3
…
27 28