How to create GoogleCredential object referencing the service account json file in Dataflow?

Question

I have written a pipeline to extract G suite activity logs by referring the G suite java-quickstart where the code reads client_secret.json file as below,

InputStream in = new FileInputStream("D://mypath/client_secret.json");
GoogleClientSecrets clientSecrets = GoogleClientSecrets.load(JSON_FACTORY, new InputStreamReader(in));

Pipeline runs as expected in local(runner=DirectRunner) but the same code fails with java.io.FileNotFoundException expection when executed on cloud(runner=DataflowRunner)

I understand local path is invalid when executed on cloud. Any suggestion here?

Updated:

I have modified the code as below and I am able to read the client_secrets.json file

    InputStream in =
    Activities.class.getResourceAsStream("client_secret.json");

Actual problem is in creating the credential object

private static   java.io.File DATA_STORE_DIR = new java.io.File(System.getProperty("user.home"),
         ".credentials/admin-reports_v1-java-quickstart");
private static final List<String> SCOPES = Arrays.asList(ReportsScopes.ADMIN_REPORTS_AUDIT_READONLY);

static {
    try {
        HTTP_TRANSPORT = GoogleNetHttpTransport.newTrustedTransport();
        DATA_STORE_FACTORY = new FileDataStoreFactory(DATA_STORE_DIR);
    } catch (Throwable t) {
        t.printStackTrace();
        System.exit(1);
    }
}

public static Credential authorize() throws IOException {
    // Load client secrets.
    InputStream in =
    Activities.class.getResourceAsStream("client_secret.json");

    GoogleClientSecrets clientSecrets = GoogleClientSecrets.load(JSON_FACTORY, new InputStreamReader(in));

    // Build flow and trigger user authorization request.
    GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow.Builder(HTTP_TRANSPORT, JSON_FACTORY,
            clientSecrets, SCOPES).setDataStoreFactory(DATA_STORE_FACTORY).setAccessType("offline").build();
    Credential credential = new AuthorizationCodeInstalledApp(flow, new LocalServerReceiver()).authorize("user");
    System.out.println("Credentials saved to " + DATA_STORE_DIR.getAbsolutePath());
    return credential;
}

Observations:

Local execution:

1. On initial execution, program attempts to open browser to authorize the request and stores the authenticated object in a file - "StoredCredential".
2. On further executions, the stored file is used to make API calls.

Running on cloud(DataFlowRunner):

1. When I check logs, dataflow tries to open a browser to authenticate the request and stops there.

What I need?

How to modify GoogleAuthorizationCodeFlow.Builder such that the credential object can be created while running as dataflow pipeline?

Answer 1

I have found a solution to create GoogleCredential object using the service account. Below is the code for it.

    public static Credential authorize() throws IOException, GeneralSecurityException {

        String emailAddress = "service_account.iam.gserviceaccount.com";
        GoogleCredential credential = new GoogleCredential.Builder()
                .setTransport(HTTP_TRANSPORT)
                .setJsonFactory(JSON_FACTORY)
                .setServiceAccountId(emailAddress)
                .setServiceAccountPrivateKeyFromP12File(Activities.class.getResourceAsStream("MYFILE.p12"))
                .setServiceAccountScopes(Collections.singleton(ReportsScopes.ADMIN_REPORTS_AUDIT_READONLY))
                .setServiceAccountUser("USER_NAME")
                .build();

        return credential;
    }

Answer 2

Can you try running the program multiple times locally. What I am wondering is, if the "StoredCredential" file is available, will it just work? Or will it try to load up the browser again?

If so, can you determine the proper place to store that file, and download a copy of it from GCS onto the Dataflow worker? There should be APIs to download GCS files bundled with the dataflow SDK jar. So you should be able to use those to download the credential file.