Stack Overflow
Stack Overflow

Questions tagged [aws-batch]

AWS Batch enables you to run batch computing workloads on the AWS Cloud. Batch computing is a common way for developers, scientists, and engineers to access large amounts of compute resources. AWS Batch removes the undifferentiated heavy lifting of configuring and managing the required infrastructure.

420 questions
0
votes
1 answer

How to achieve Parallelism using AWS Batch Multi-node Parallel Jobs

I've got an SQS queue that will be filled with a json message when my S3 bucket has any CREATE event. Message contains bucket and object name Also have Docker image which contains python script that will read message from sqs. With help of that…
Sarathy Velmurugan
  • 123
  • 1
  • 2
  • 10
0
votes
0 answers

AWS Batch Boto3 Change Job_ID

I am running a boto3 batch client job and using the submit_job function like so: submit_job_response = batch.submit_job( jobName=job_name, jobQueue=job_queue, jobDefinition=job_definition, parameters=parameters, …
Alex
  • 280
  • 3
  • 18
0
votes
1 answer

get instance type from AWS managed batch job

I'm using a step function to launch a batch job, how do I get the instance type that was ultimately used to process my job in a compute environment that is AWS managed? It doesn't look like this information is available with the CLI command…
Constantino
  • 2,243
  • 2
  • 24
  • 41
0
votes
0 answers

AWS ETL solutions for small data

My objective is to get the data from S3 files, transform and save it to a datasource(could be dynamoDB or RDS). And the filesize would be <20MB and there could be multiple(~10) such files uploaded periodically (once a day). I'm considering using…
user3035692
  • 11
  • 3
0
votes
0 answers

AWS Batch cannot create c5d.18xlarge instance type

I'm trying to submit job with c5d.18xlarge (vcpus: 72, memory: 144000) but it never coming up till 20 mins even my maximum vcpus of compute enviroment is 256? let containerInfo = { jobDefinition: '', jobName: '', jobQueue: , …
PPShein
  • 13,309
  • 42
  • 142
  • 227
0
votes
1 answer

AWS ECS DescribeContainerInstances generates InvalidParameterException because of accountId mismatch

From a container job running in Batch, I'm trying to extract the instance id of the EC2 instance that is running that AWS batch job: I'm proceeding as follows: In the job script, I grab the environment variable AWS_BATCH_JOB_ID and call…
init_js
  • 4,143
  • 2
  • 23
  • 53
0
votes
1 answer

Selenium failing on AWS-batch

I am running a python script that implements selenium and geckodriver to login to a website account. The code works locally in a container with no problems, however when I deploy it to AWS-batch it stops working reliably. I run this job at a…
Stan
  • 1
  • 1
0
votes
1 answer

I am trying to invoke a batch operation through AWS lambda. I want the batch to return the output to my lambda

I am invoking a batch job through AWS Lambda. The output can be seen in the cloudwatch log, however I want the output of AWS batch to be shown in the lambda logs as I intend to integrate an API with lambda. Lambda being my source of…
Abhinav Bhardwaj
  • 1
  • 1
0
votes
1 answer

api gateway integration with aws batch

I want to use API gateway with AWS batch I already know how to use API Gateway with AWS lambda naad since there is a limit of 250 MB in lambda, I am not able to use it for integration and now trying AWS batch
arpit agrawal
  • 1
  • 1
  • 1
0
votes
1 answer

AWS-Batch vs EC2 vs AWS Workspaces for running batch scripts to load data to Redshift

I have multiple CSV files containing data for different tables, with different file sizes varying from 1 MB to 1.5 GB. I want to process the data (replace/remove values of columns) row by row and then load the data to existing Redshift tables. This…
Swapnil
  • 125
  • 1
  • 9
0
votes
1 answer

How to put Job Status in the Subject of the email that SNS sends

I am running an AWS Batch job as Cloudwatch Event Target and I am using SNS to send a notification email about the status of the job. I have created a SNS Topic, subscribed it to my email address and set it up as a target it a Cloudwatch event…
Seeker90
  • 785
  • 4
  • 17
  • 37
0
votes
1 answer

How do you set a Content-Type Header for the Ruby AWS SDK?

I'm using Webmock to test http requests made by the ruby aws-sdk (in this case aws-sdk-batch). Webmock handles json requests with a much more convenient hash diff and partial matching when the requests fail, but it will only do so if the…
Jacob Dalton
  • 1,643
  • 14
  • 23
0
votes
1 answer

AWS Batch: How do we know which compute environment ordering a batch job is using

I am new to AWS .I have two compute environments. When I submit my batch JOB how do I know which compute environment is it using? I don't see anything on Batch job logs.
Rohan Nayak
  • 233
  • 4
  • 14
0
votes
2 answers

AWS Batch - passing UserData to Container Instances

I am working on aws batch to pass user data during the initialization of the container instance. Is there a way to do it in aws batch ? Followed this link Create AWS Batch Managed Compute Environment passing UserData to Container Instances but not…
sara
  • 41
  • 6
0
votes
1 answer

Why python tesserocr not using 4 CPU cores on AWS Batch?

I'm trying to get tesserocr python library to run on 4 cores. According to tesseract docs, I understand it supports up to 4 cores. I have a tesserocr python3.x job running inside AWS Batch (docker container based on amazonlinux:latest image) on a…
Sagi Mann
  • 2,967
  • 6
  • 39
  • 72
1 2 3
…
27 28