Questions tagged [azure-batch]

The azure-batch tag should be used to ask questions regarding all aspects of the Azure Batch service including job scheduling, compute node provisioning, and pool orchestration. Extended ecosystem questions are also welcome that are related to Azure Batch accelerator offerings, Azure Portal Batch blades, and Batch Explorer.

Azure Batch is a job scheduling and compute management orchestration solution for the Microsoft Azure cloud. The service is available in all Azure regions and can deploy with nearly all Azure VM sizes available, including those with specialized hardware such as GPUs and Infiniband/RDMA interfaces.

Azure Batch is a managed job scheduling solution where the underlying compute resources are allocated as pools. Pools may be comprised of Linux or Windows VMs and any number of dedicated and low priority nodes and can be automatically scaled through autoscale formulas. An Azure Batch Job can be thought of as a queue for tasks. Each Azure Batch Job can be assigned to a pool. Any number of tasks can be assigned to jobs, including multi-instance tasks (i.e., multi-node) which support MPI jobs. Azure Batch also supports containerized execution.

For a full introductory treatment of Azure Batch, see the Azure Batch Technical Overview. The service team also provides a GitHub repository for issue tracking, feature requests, and change logs.

416 questions
1
vote
1 answer

Azure batch start task exceeds the autoScaleEvaluationInterval

I am working on Azure batch and facing an issue. I have configured a start task on my pool with wait for success as true as the start task should complete before starting the job task. I also want to set the autoscale formula on my pool. I want to…
Tarun
  • 271
  • 1
  • 6
  • 18
1
vote
2 answers

How to use batch account created with user subscription in Data Factory Linked Service?

I have created a Batch Service with user subscription and created a key vault also, http://csharpcorner.mindcrackerinc.netdna-cdn.com/article/de/Images/image007.jpg Now how can I use that batch service in the Data Factory Linked Service. Because it…
Arron
  • 1,134
  • 2
  • 13
  • 32
1
vote
1 answer

Azure Batch - problems with Active Directory authentication

I'm following this other post about creating Azure Batch pools with custom machine images. I'm stuck on the Azure Active Directory (AD) authentication. I've followed all the tutorials on the other thread for registering an application and obtaining…
RedPanda
  • 522
  • 6
  • 15
1
vote
3 answers

Access Azure Batch from an Azure Function

I'm trying to use a Service Principle to access a Batch pool from an Azure Function and running into authentication issues that I don't understand. The initial login with the Service Principle works fine, but then using the credentials to access the…
Mark
  • 816
  • 1
  • 9
  • 27
1
vote
2 answers

Task failure and Exit code -532462766 in Azure Batch

I am trying to run a Console Application in Azure Batch. I have created the Client Application having responsibility of creating and running Job & Tasks in the Azure Batch so far. My ClientApplication successfully creates a Job, a Task within the…
starklord
  • 291
  • 2
  • 6
  • 16
1
vote
1 answer

How to deploy an application that uses SqlServer as db server to Azure Batch

I am trying to execute a pre-existing application on Azure Batch. What I have done so far that I have studied about Azure Batch, since I am a beginner for Azure, and I have created a sample application in C# .NET (Console Application) that…
starklord
  • 291
  • 2
  • 6
  • 16
1
vote
0 answers

Azure Batch BlobNotFound error in pool start task

I'm attempting to create a Batch pool of workers with a 'Start task' that that uses the Start task's 'Resource files' to download an application tar from blob storage and install it. I've placed the tar file in the blob storage account associated…
1
vote
1 answer

Extracting archives with azure functions

I'm currently using azure functions for the following process : Dowload tar.gz archives from a client ftp. Most of them are daily packages under 20mo, but a few of them are more than 100mo (edited once year ends and replace all previous…
Pierre Jeannet
  • 255
  • 1
  • 10
1
vote
1 answer

Combine 65000 small xml files in blobs storage and copy them into data lake as big CSV files

I have around 65000 small xml files (around 1 to 3 kb per file) per hour in Azure blob storage. Its telemetry data and will be there in blob each hour. I want to combine them all and create big xml files(hourly one file or so) and convert that into…
1
vote
1 answer

Azure Batch Linked Service Not Found in ADF Pipeline

I am trying to deploy an ADF pipeline using an ARM template. Below is the portion of the code for by azure batch linked service: { "type": "linkedservice", "name": "AzureBatchLS", "dependsOn": [ …
1
vote
1 answer

Using Windows Containers directly in Azure Batch

I have a Docker-based Windows Container image, packaged to facilitate a very specific execution environment. It also contains the application that will run automatically when started. I have a workload to process, that varies in size (from a 100…
Web User
  • 7,438
  • 14
  • 64
  • 92
1
vote
1 answer

Timeout when deleting Azure Batch certificate

I receive the following in the portal: There was an error while deleting [THUMBPRINT HERE]. The server returned 500 error. Do you want to try again? I suspect that there is an azure batch pool / node hanging on to the certificate, however the…
Alex KeySmith
  • 16,657
  • 11
  • 74
  • 152
1
vote
1 answer

Custom .NET activity to run executable and copy output files to blob from Azure Batch

I am trying to create a custom .NET activity in Azure, which creates a pool of VMs and a job to process the extraction of files on the pool nodes. So far I have the code for all this, but I am not sure how to download the processed file from the…
FeodorG
  • 178
  • 2
  • 10
1
vote
1 answer

Task exited with exit code null

I am running into a problem of a task exiting with a null exit code. And with this exit code, I noted that I can't access files on node to check the stderr and stdout files. What could be the problem? Also, what does a null exit code mean and how…
J.B
  • 137
  • 2
  • 16
1
vote
1 answer

When running Azure Data Factory Activities in Azure Batch, how should asynchronicity be handled

Background I have somewhat simplified this scenario but this is the general problem. I am using an Azure Data Factory to ingest data from a custom API into a table in Azure Data Warehouse. I am using an IDotNetActivity to run the C# code that calls…