Download files from Google Storage using Java

Question

I have successfully automated the process to move data from Google Big Query, to Google Storage. Now I need to download the data from Google Storage to my environment in an automated way as well.

I am trying to do a normal HTTP request, but authorizing before. So my HTTP request is

    HttpRequestFactory requestFactory = HTTP_TRANSPORT.createRequestFactory(authorize());
    GenericUrl url = new GenericUrl(uri);
    HttpRequest request = requestFactory.buildGetRequest(url);
    HttpResponse response = request.execute();
    String content = response.parseAsString();

And my authorization code is

/** Authorizes the installed application to access user's protected data. */
    private static Credential authorize() throws Exception
    {
        HttpTransport httpTransport = new NetHttpTransport();
        JsonFactory jsonFactory = new JacksonFactory();

        // load client secrets
        GoogleClientSecrets clientSecrets = GoogleClientSecrets.load(JSON_FACTORY,
                new InputStreamReader(BigQueryConsumer.class.getResourceAsStream("/secret.json")));

        // This creates the credentials datastore at ~/.oauth-credentials/${credentialDatastore}
        FileDataStoreFactory fileDataStoreFactory = new FileDataStoreFactory(new File(System.getProperty("user.home") + "/" + CREDENTIALS_DIRECTORY));

        // set up authorization code flow
        GoogleAuthorizationCodeFlow flow = new GoogleAuthorizationCodeFlow.Builder(
                httpTransport, JSON_FACTORY, clientSecrets,
                SCOPES).setDataStoreFactory(fileDataStoreFactory)
                .build();
        // authorize
        return new AuthorizationCodeInstalledApp(flow, new LocalServerReceiver()).authorize("user");
    }

Where the following constants are

CREDENTIALS_DIRECTORY : ".oauth-credentials"
JSON_FACTORY : JacksonFactory.getDefaultInstance()
SCOPES : A list of string having just "https://www.googleapis.com/auth/devstorage.full_control"
HTTP_TRANSPORT : new NetHttpTransport()

What am I missing during the authentication/authorization process? I am getting

    Exception in thread "main" com.google.api.client.http.HttpResponseException: 401 Unauthorized
<HTML>
<HEAD>
<TITLE>Unauthorized</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>Unauthorized</H1>
<H2>Error 401</H2>
</BODY>
</HTML>

You might also want to try out [`gcloud-java`](http://googlecloudplatform.github.io/gcloud-java/), here's some [sample code](https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-examples/src/main/java/com/google/gcloud/examples/StorageExample.java). — Misha Brukman, Oct 20 '15 at 15:51

Mez · Answer 1 · 2015-10-19T21:56:00.857

For others looking for an answer. There are several steps that you need to follow in order to use the application default credentials... You can follow the steps below, or have a look at this link.

First things first : install google cloud storage SDK
Make sure you can run commands from the SDK. If you do not have python installed you need to install python 2.7 or above, and also pyopenssl...
You need to authenticate with from within the SDK by running the gcloud auth activate-service-account [Service account email] --key-fil e [.p12 file] (Edit the values in the square brackets). When you run this you should get a message telling you that you have activated your service account
You need to set environment variables from the SDK by setting GOOGLE_APPLICATION_CREDENTIALS to the JSON path of the secret (the one downloaded from the service account created from the developers console) CLOUDSDK_PYTHON_SITEPACKAGES to 1, and also setting the project

Commands to configure system variables...

set GOOGLE_APPLICATION_CREDENTIALS "secret.json path"
set CLOUDSDK_PYTHON_SITEPACKAGES 1
gcloud config set project "your project name"

After you authenticate, and authorize yourself, you can then start using the applications default credentials, given that you have setup your environment properly.

When you successfully setup your environment, you can then authenticate and get default credentials by just

        GoogleCredential credential = GoogleCredential.getApplicationDefault();

        if (credential.createScopedRequired()) {
            credential = credential.createScoped(StorageScopes.all());
        }

        HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
        storageService = new Storage.Builder(httpTransport, jsonFactory, credential)
                .setApplicationName(applicationName).build();

And using HTTP GET to get objects from your google storage bucket, i.e. something like the following

// Set up and execute a Google Cloud Storage request.
                String uri = "https://storage.googleapis.com/" + URLEncoder.encode("[your bucket name here]", "UTF-8") + "/" + googleStorageFileName + "/" + fileName;

                httpTransport = GoogleNetHttpTransport.newTrustedTransport();
                HttpRequestFactory requestFactory = httpTransport.createRequestFactory(
                        credential);
                GenericUrl url = new GenericUrl(uri);

                HttpRequest request = requestFactory.buildGetRequest(url);
                HttpResponse response = request.execute();
                String content = response.parseAsString();
                BufferedWriter writer = new BufferedWriter( new FileWriter( pathToSave + googleStorageFileName));
                writer.write( content);

Download files from Google Storage using Java

1 Answers1