How to submit apache beam dataflow job to GCP through java application

Question

I have a dataflow job which is written in apache beam with java. I am able run the dataflow job in GCP through this steps.

Created dataflow template from my code. Then uploading template in cloud storage.
Directly creating job from template option available in GCP->Dataflow->jobs This flow is working fine. I want to do same step through java app. means, I have one api when someone sends request to that api, I want to start this dataflow job through the template which I have already stored in storage.

I could see rest api is available to implement this approach. as below,

POST /v1b3/projects/project_id/locations/loc/templates:launch?gcsPath=template-location

But I didn't find any reference or samples for this. I tried the below approach In my springboot project I added this dependency

<!-- https://mvnrepository.com/artifact/com.google.apis/google-api-services-dataflow -->
        <dependency>
            <groupId>com.google.apis</groupId>
            <artifactId>google-api-services-dataflow</artifactId>
            <version>v1b3-rev20210825-1.32.1</version>
        </dependency>

and added below code in controller

public static void createJob() throws IOException {
        GoogleCredential credential = GoogleCredential.fromStream(new FileInputStream("myCertKey.json")).createScoped(
            java.util.Arrays.asList("https://www.googleapis.com/auth/cloud-platform"));       
        try{
            Dataflow dataflow = new Dataflow.Builder(new LowLevelHttpRequest(), new JacksonFactory(),
            credential).setApplicationName("my-job").build();           --- this gives error
            
            //RuntimeEnvironment
            RuntimeEnvironment env = new RuntimeEnvironment();
            env.setBypassTempDirValidation(false);
            //all my env configs added

            //parameters
            HashMap<String,String> params = new HashMap<>();
            params.put("bigtableEmulatorPort", "-1"); 
            params.put("gcsPath", "gs://bucket//my.json");
            // all other params
            
            LaunchTemplateParameters content = new LaunchTemplateParameters();
            content.setJobName("Test-job");
            content.setEnvironment(env);
            content.setParameters(params);
            
            dataflow.projects().locations().templates().launch("project-id", "location", content);           
        }catch (Exception e){
            log.info("error occured", e);
        }
    }

This gives {"id":null,"message":"'boolean com.google.api.client.http.HttpTransport.isMtls()'"}

error in this line itself

Dataflow dataflow = new Dataflow.Builder(new LowLevelHttpRequest(), new JacksonFactory(),
            credential).setApplicationName("my-job").build();

this is bcs, this dataflow builder expects HttpTransport as 1st argument but I passed LowLevelHttpRequest()

I am not sure is this the correct way to implement this. Can any one suggest any ideas on this? how to implement this? any examples or reference ?

Thanks a lot :)

How to submit apache beam dataflow job to GCP through java application

0 Answers0