Stack Overflow
Stack Overflow

Questions tagged [aws-batch]

AWS Batch enables you to run batch computing workloads on the AWS Cloud. Batch computing is a common way for developers, scientists, and engineers to access large amounts of compute resources. AWS Batch removes the undifferentiated heavy lifting of configuring and managing the required infrastructure.

420 questions
6
votes
1 answer

AWS Step cannot correctly invoke AWS Batch job with complex parameters

I have an existing AWS Steps orchestration that is executing a AWS Batch job via lambdas. However AWS have recently added the ability to directly invoke other services like AWS Batch from a step. I am keen to use this new functionality but cannot…
Andrew Patterson
  • 1,338
  • 1
  • 12
  • 26
6
votes
2 answers

How do I retrieve AWS Batch job parameters?

How do I retrieve parameters from an AWS Batch job request? Suppose I have a job submitter app that sends a job request with the following code (in C#): SubmitJobRequest submitJobRequest = new SubmitJobRequest() { JobName =…
sevencontinents
  • 173
  • 2
  • 8
6
votes
5 answers

Is there a way to set a walltime on AWS Batch jobs?

Is there a way to set a maximum running time for AWS Batch jobs (or queues)? This is a standard setting in most batch managers, which avoids wasting resources when a job hangs for whatever reason.
static_rtti
  • 53,760
  • 47
  • 136
  • 192
6
votes
2 answers

Limits for AWS Batch job details retention

I'm trying to understand how long the details associated with an AWS Batch job are retained. For example, the Kinesis limits page describes how each stream defaults to a 24 hour retention period that is extendable up to 7 days. The AWS Batch limits…
Matt Lavin
  • 932
  • 10
  • 21
5
votes
2 answers

Starting AWS Batch from SQS queue

I have a long running batch job that am planning to move to AWS Batch. Since it takes couple of hours to run this job, I cant use AWS Lambda. But, one of the requirement is to trigger this job from my AWS SQS queue. I couldn't find any reference…
Ashish Joseph
  • 1,103
  • 3
  • 12
  • 35
5
votes
3 answers

AWS - Batch vs Fargate

I have a docker image. I would like to create a container periodically and execute as a job, say every 1 hour, by creating CloudWatch Rule. As we are using AWS cloud, I am looking at the AWS Batch service. Interestingly there is also a ECS Scheduled…
RamPrakash
  • 2,218
  • 3
  • 25
  • 54
5
votes
2 answers

AWS CDK StateMachine BatchSubmitJob with dynamic environment variables

I'm trying to create a statemachine with a BatchSubmitJob in AWS CDK with dynamic environment variables in the BatchContainerOverrides. I was thinking about something like this: container_overrides = sfn_tasks.BatchContainerOverrides( …
Daphne
  • 63
  • 6
5
votes
3 answers

AWS Batch vs Spring Batch

I have been planning to migrate my Batch processing from Spring Batch to AWS Batch. Can someone give me the reasons to Choose AWS Batch over Spring Batch?
Riley
  • 51
  • 1
  • 2
5
votes
1 answer

Docker image in AWS Step Functions

Let's suppose I have an AWS Step function, a state of which is a Batch Job, associated with a Docker image. Is there more efficient way to associate a Docker image with a state, instead of using a Batch Job (arn:aws:states:::batch:submitJob.sync)?…
anton
  • 247
  • 3
  • 13
5
votes
1 answer

AWS Batch Job stuck RUNNABLE when Launch template is configured

I have configured Step Function with AWS Batch Jobs. All configuration working well but I need to customize starting instance. For this purpose I use Launch Template service and build simple (empty) configuration based on instance type used in AWS…
Geo ZiDani
  • 111
  • 1
  • 1
  • 8
5
votes
0 answers

AWS batch Fargate ECS container has no internet connectivity in public subnet

I have deployed an AWS batch job which creates an ECS Fargate task in the background. This is all deployed in a public subnet with internet access. I have verified that running the docker container in a standalone EC2 instance in the same subnet has…
medium
  • 4,136
  • 16
  • 55
  • 66
5
votes
1 answer

AWS Batch job start too long with Min vCPUs=0 in Compute environment

I`m using AWS Batch. After job submitting I wait 10-15 minutes until my job gets RUNNING status. My Compute environment configuration is next: Provisioning model: EC2 Instance types: m4.xlarge Min vCPUs: 0 Desired vCPUs: 0 Max vCPUs: 4 ECR image…
Valeriy K.
  • 2,616
  • 1
  • 30
  • 53
5
votes
2 answers

Create AWS Batch Managed Compute Environment passing UserData to Container Instances

I would like to create a Managed Compute Environment for AWS Batch, but use EC2 User Data to configure the instances as they are brought into the ECS fleet that Batch is scheduling jobs onto. It shouldn't matter, but the purpose of the User Data…
mcg256
  • 150
  • 1
  • 6
5
votes
1 answer

AWS Batch failover strategy

I am using AWS batch for executing jobs, I am calculating the initial memory to use by content size. About 90% of times its successful but 10% times it fails with OutOfMemory error. So for next attempt for this failed jobs, I would like to increase…
mightymahesh
  • 303
  • 3
  • 14
4
votes
1 answer

What is the difference between AWS Batch and Sagemaker Training Job for Machine Learning training

What is the difference between AWS Batch and Sagemaker Training Job when using it for running docker image for Machine Learning training?
ryfeus
  • 323
  • 2
  • 6
1 2
3
…
27 28