How to use AWS batch in java class(lambda function) to submit batch job

Question

I read the output from stepfunctions using envContainerOverrides and then I am calling my batch job with this nodejscode: Here I am reading the environment variables which is passed to the batchjob.

Stepfunctions output:

{"bucketName":"bucketName","filesList":["filelist-bucket/filelist1.txt","filelist-bucket/filelist2.txt"]}

This nodejs code is not able to fetch if its an array [filelist-bucket/filelist1.txt","filelist-bucket/filelist2.txt] it works perfectly if its a single value.

I want to convert this nodejs code into java :

'use strict';

const AWS = require('aws-sdk');

console.log('Loading function');

exports.handler = (event, context, callback) => {
    // Take the data from step 1 and modify, send to standard output
    var comment = event.Comment ;

    var envContainerOverrides ={
        "environment" :[
          {
           "name":"s3Bucket",
           "value":event.bucketName
          },
          {
           "name":"s3FileList",
           "value":event.filesListUrl
          }
        ]
    };
    const params = {
        jobDefinition: process.env.JOB_DEFINITION,
        jobName: process.env.JOB_NAME,
        jobQueue:process.env.JOB_QUEUE,
        containerOverrides: envContainerOverrides || null,
        parameters: event.parameters || null,
    };


    // Submit the Batch Job
    new AWS.Batch().submitJob(params, (err, data) => {
        if (err) {
            console.error(err);
            const message = `Error calling SubmitJob for: ${event.jobName}`;
            console.error(message);
            callback(message);
        } else {
            const jobId = data.jobId;
            console.log('jobId:', jobId);
            callback(null, "Job Id : "+jobId);
        }
    });



};

I am doing something like this:

public class InitiateBatchJob1 {

    public static BatchJobRequest process(BatchJobRequest batchJobRequest) throws Exception {


        String s3Bucket = batchJobRequest.getBucketName();
        List<String> s3FileList = batchJobRequest.getFilesListUrl();

        Job job = new Job();
        job.setJobDefinition("testbatchjobenv:2");
        job.setJobQueue("nbatchjobqueue");
        job.setJobName("Filedownload");


        /*// Submit the Batch Job
        new AWS.Batch().submitJob(params, (err, data) => {
            if (err) {
                console.error(err);
                const message = `Error calling SubmitJob for: ${event.jobName}`;
                console.error(message);
                callback(message);
            } else {
                const jobId = data.jobId;
                console.log('jobId:', jobId);
                callback(null, "Job Id : "+jobId);
            }
        });
*/



        return null;
    }


}

BatchJobRequest.java

import java.util.List;

public class BatchJobRequest {

    private String bucketName;

    private List<String> filesListUrl;

    public String getBucketName() {
        return bucketName;
    }

    public void setBucketName(String bucketName) {
        this.bucketName = bucketName;
    }

    public List<String> getFilesListUrl() {
        return filesListUrl;
    }

    public void setFilesListUrl(List<String> filesListUrl) {
        this.filesListUrl = filesListUrl;
    }

}

Job.java:

public class Job {

    private String jobDefinition;
    private String jobName;
    private String jobQueue;
    private String containerOverrides;
    private String parameters;
    public String getJobDefinition() {
        return jobDefinition;
    }
    public void setJobDefinition(String jobDefinition) {
        this.jobDefinition = jobDefinition;
    }
    public String getJobName() {
        return jobName;
    }
    public void setJobName(String jobName) {
        this.jobName = jobName;
    }
    public String getJobQueue() {
        return jobQueue;
    }
    public void setJobQueue(String jobQueue) {
        this.jobQueue = jobQueue;
    }
    public String getContainerOverrides() {
        return containerOverrides;
    }
    public void setContainerOverrides(String containerOverrides) {
        this.containerOverrides = containerOverrides;
    }
    public String getParameters() {
        return parameters;
    }
    public void setParameters(String parameters) {
        this.parameters = parameters;
    }


}

and I have found this api for AWSBatch http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/batch/AWSBatch.html#cancelJob-com.amazonaws.services.batch.model.CancelJobRequest-

But not sure if its the correct api to use in my class.I searched online there isnt much links on using aws batch in java.

Basically I need to know how to set job denfition,jobName,job queue and submit batchjob using java class .Can anyone please help me with this.

Answer 1

You have to do something like this: http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/batch/AWSBatch.html#submitJob-com.amazonaws.services.batch.model.SubmitJobRequest-

AWSBatch client = AWSBatchClientBuilder.standard().build();
    SubmitJobRequest request = new SubmitJobRequest().withJobName("some-name")
            .withJobQueue("job-queue-name")
            .withJobDefinition("job-definition-name-with-revision-number:1");
    SubmitJobResult response = client.submitJob(request);

I used this code with the lambda function and the IAM role with AWSBatchFullAccess that I created in the aws console. So after building and loading jar on the aws, my 'client' was initialized using the data taken from lambda. In your application, it seems like you also need to add data to initialize the client. You have to use the method

AWSBatch build(AwsSyncClientParams params)

look AWSBatchClientBuilder class.

You still need to create Job queue and Job definition. Also I would advise you to start working with the aws console. Here is good tutorials to understand aws batch workflow http://technology.finra.org/code/enjoying-auto-scaling-integrated-authentication-low-host-cost.html http://www.awsomeblog.com/analysing-exif-data-with-aws-batch/. After this tutorial you can add code for submitting job in myHandler method and see how job starts.