AWS Batch enables you to run batch computing workloads on the AWS Cloud. Batch computing is a common way for developers, scientists, and engineers to access large amounts of compute resources. AWS Batch removes the undifferentiated heavy lifting of configuring and managing the required infrastructure.
Questions tagged [aws-batch]
420 questions
0
votes
1 answer
Metaflow explanation on how GPU is utilized
Objective
Understand how GPU will be utilized in Metaflow.
Background
As in Documentation / Explanation on how to use GPU #250, there are several discussions on how to use GPU.
It looks @resources(GPU=2) looks after the GPU allocation, but there are…
mon
- 18,789
- 22
- 112
- 205
0
votes
0 answers
Using a local Docker container to call out to AWS batch and sumbit jobs and post data after jobs are complete
I have a local dev environment running Docker containers. I have since migrated to using AWS batch to run these jobs and post results in my 'prod' environment. I would like to mirror this functionality locally. Essentially set up an endpoint that my…
BucksDad
- 15
- 3
0
votes
1 answer
Compute environments are not displayed in AWS console
Compute environments created via boto3 are not displayed in AWS console. I can see them in the batch_client.describe_compute_environments() call response:
{
'computeEnvironmentName': 'name',
'computeEnvironmentArn':…
Alex Barysevich
- 544
- 1
- 7
- 18
0
votes
1 answer
How to schedule or trigger a SQL procedure in AWS Aurora RDS?
I have a situation where data is getting loaded into staging tables of Aurora MySQL from AWS S3 using Lambda function on a daily basis. Now, I need to schedule a job in Aurora DB which will trigger a SQL Procedure (having some transformation logic)…
Nehal
- 1
- 2
0
votes
0 answers
Temporary s3 buckets or different storage methodologies to support AWS batch execution
I want to pass data to and from an AWS batch instance.
The current workflow requires an s3 bucket as support, it requires to take care of s3 data upload and disposal and to create an existing bucket just for this data exchange.
Is it possible to…
gc5
- 9,468
- 24
- 90
- 151
0
votes
1 answer
AWS Step Functions invoke Dynamic Parallel Step Functions
I'm solving a problem using step function workflows.
The problem goes like this, I have a worklow of 10 AWS Batch jobs.
The first 3 jobs run in sequence and the 4-7 jobs are dynamic steps i.e, they needs to run multiple times with different…
user12073121
- 81
- 3
- 11
0
votes
0 answers
How to configure a Security Group to allow an ECS container instance to reach Aurora RDS in a VPC?
Using Terraform I have configured a batch job definition, job queue, and compute environment that allows for job submissions on AWS Batch. The job launches an ECS container that runs a Python script which makes a connection to an Aurora database…
James Adams
- 8,448
- 21
- 89
- 148
0
votes
0 answers
Invoking concurrent AWS Batch jobs for a group of SQS messages
I have millions of independent tasks in the SQS FIFO queue. Each task can take 1-3 minutes to get completed and needs to update status in RDS.
I want to take a batch of say 10,000 messages (tasks) and run it in an AWS batch job. I want to run all…
Dev
- 13,492
- 19
- 81
- 174
0
votes
1 answer
AWS Batch Service taking longer time to launch instance
I am observing that if a new batch job has been submitted shortly after the last instance in a compute environment has shut down it takes over 10 minutes for Batch to add a new instance to the Compute environments even if CE's have available…
Chaitanya Vats
- 41
- 1
- 5
0
votes
2 answers
AWS Schedule Batch-Job via Boto3 SDK with pass some environment variable
i have one AWS Batch-Job which is working fine, the scenario is when we want to run the Job we are submitting Batch-Job using python BOTO3 SDK.
Now I want to schedule the Batch-Job in which I can specify the time and it needs to submit Batch-Job on…
Maulik Harkhani
- 390
- 3
- 12
0
votes
1 answer
Best Option(s) for Running Python ETL Code in AWS
I am looking for a recommendation on which AWS service (or combination thereof) to use to execute an ETL code in Python to transform text-based files:
Description of the code/process:
1. Python code transforms input text files from a custom vendor…
bda
- 372
- 1
- 7
- 22
0
votes
1 answer
How do I correctly override a decorator function in Luigi for AWS Batch?
I’m using Luigi to kick off an AWS Batch job.
I want to create a subclass of luigi.contrib.batch.BatchTask (the documentation for Luigi for AWS Batch (relies on boto3) can be found here:…
MrFronk
- 382
- 5
- 23
0
votes
2 answers
Aws lambda vs aws batch
I am currently working on a project where I need to merge two significantly large csv files into one(both are a few hundred MBs). I am fairly new to aws. I am aware of memory allocation and execution time limitations of lambda. Other than that are…
user2803056
- 377
- 4
- 12
0
votes
0 answers
Understanding Batch Job Behavior
If we are creating 100 batch jobs at a time and each batch job will take around 2 mins for say. Then i need to understand whether batch jobs will run Sequentially or parallelly. If Sequentially , can i say we need to wait for the previous job to…
Chaitanya Vats
- 41
- 1
- 5
0
votes
1 answer
Docker container with Selenium and Chrome webdriver crashes when multiple containers run in parallel on AWS Batch
We are running AWS Batch jobs that launch Docker containers to run Selenium and Chrome with python 3.6. When we set it up to run multiple containers per server, jobs often start up, run for a couple minutes, then crash with chrome not reachable:
…
kielni
- 4,779
- 24
- 21