Stack Overflow
Stack Overflow

Questions tagged [aws-batch]

AWS Batch enables you to run batch computing workloads on the AWS Cloud. Batch computing is a common way for developers, scientists, and engineers to access large amounts of compute resources. AWS Batch removes the undifferentiated heavy lifting of configuring and managing the required infrastructure.

420 questions
8
votes
1 answer

How to check Memory and CPU usage of AWS Batch job

while creating a batch job, we can provide vCPUs and Memory in Container properties. In cloudwatch, we can check memory and CPU usage per cluster under Elastic Container Service per cluster. Is there any way to check memory and CPU usage per job?
Dev
  • 13,492
  • 19
  • 81
  • 174
8
votes
1 answer

How to pass script arguments to AWS Batch fetch-and-run

I am following this tutorial to run a simple fetch-and-run example in AWS batch. However, I'm unable to pass arguments to the script fetched through this example. The basic example will produce this execution: export BATCH_FILE_TYPE="script" export…
gc5
  • 9,468
  • 24
  • 90
  • 151
8
votes
2 answers

Using AWS Batch can a docker image be specified dynamically in a job definition?

I want to create jobs in AWS Batch that vary on the image that is used to launch the container. I'd like to do this without creating a different Job Definition for each image. Is it possible to parameterize the image property using job definition…
Segfault
  • 8,036
  • 3
  • 35
  • 54
8
votes
2 answers

AWS Batch Arrays - array size?

I'm using AWS Batch and have started using Array Jobs. AWS_BATCH_JOB_ARRAY_INDEX is passed as an Environment Variable to the container. Is the array size passed in some way? It is mandatory to know whether the index was related to 5 jobs or 1000…
codequestions
  • 111
  • 5
8
votes
3 answers

AWS CloudWatch Rule returns FailedInvocation with AWS batch as Target

Hi I've scheduled a Cloudwatch rule in order to run every Wednesday at 14.15 GTM by having as target an AWS Batch, which always returns FailedInvocation. I'm seeing the FailedInvocation event from associated metrics However there are no logs…
Sashimi
  • 103
  • 1
  • 9
8
votes
2 answers

How can I use docker compose on AWS Batch?

I have a multi-container (docker compose) application. I would like to scale it offline on AWS Batch for processing large volumes of data on S3. My .yml file for docker compose looks something like this: version: '2' services: container1: …
user3401257
  • 81
  • 6
8
votes
2 answers

aws batch: submit job using lambda

Context: AWS, S3, Lambda, Batch. I have a lambda that is triggered when a file is uploaded in a S3 Bucket. I want that the lambda submit a Batch job. (edit: Between S3 and Lambda everything works fine. The problem is between Lambda and Batch.) Q:…
Costin
  • 2,699
  • 5
  • 25
  • 43
7
votes
0 answers

AWS : Cloudwatch logs only published after Batch job is completed

I am using a AWS Batch job , which triggers an ECR Image ( Docker Image containing python code) and the batch logs to Cloudwatch. As per the code inside Docker, I am using print command to log as shown below. The issue is all the print statements…
Sud
  • 71
  • 5
7
votes
1 answer

Mounting an elastic file system to AWS Batch Computer Enviroment

I'm trying to get my elastic file system (EFS) to be mounted in my docker container so it can be used with AWS batch. Here is what I did: Create a new AMI that is optimized for Elastic Container Services (ECS). I followed this guide here to make…
jwillis0720
  • 4,329
  • 8
  • 41
  • 74
7
votes
1 answer

AWS Lambda/ Aws Batch work flow

I have written a lambda that is triggered off s3 bucket to unzip a zip file and process a text document inside. Due to the limitation of memory of lambda i need to move my process over to something like AWS batch. Correct me if I am wrong but my…
John Hanewich
  • 167
  • 1
  • 11
6
votes
2 answers

AWS Batch Timeout connecting to ECR

I get the following error running an AWS batch job: ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s):…
alonana
  • 171
  • 2
  • 12
6
votes
0 answers

Getting Infrastructure metrics from AWS Batch to DataDog

Is there a direct integration point or a connector to integrate an AWS Batch job with DataDog? To retrieve the logs and infrastructure metrics? So far what I came across was the DataDog forwarder which can forward the logs to DataDog from Cloudwatch…
turingMan
  • 147
  • 1
  • 9
6
votes
4 answers

AWS Batch Job Execution Results in Step Function

I'm newbie to AWS Step Functions and AWS Batch. I'm trying to integrate AWS Batch Job with Step Function. AWS Batch Job executes simple python scripts which output string value (High level simplified requirement) . I need to have the python script…
pubudut
  • 603
  • 2
  • 8
  • 18
6
votes
1 answer

AWS Batch Job Stuck in Runnable State

I'm trying to run a 100 node AWS Batch job, when I set my computing environment to use only m4.xlarge and m5.xlarge instances everything works fine and my job is picked up and runs. However, when I begin to include other instance types in my compute…
Matthew Brzezinski
  • 1,685
  • 4
  • 29
  • 53
6
votes
1 answer

Why can I not override container variables for multi-node parallel jobs in AWS Batch C++ SDK?

I am working with AWS Batch. My goal is to create a multi-node parallel job through the AWS SDK for C++. For this, I have created a job definition as per the instructions here. I am working with the AWS C++ SDK, and I noticed that when I try to…
Paolo
  • 21,270
  • 6
  • 38
  • 69
1
2
3
…
27 28