Upload a file to Google Cloud Storage (Java)

Question

Basically, the issue is in the title: I cannot figure out how to upload a file to Google Storage

I have checked this webpage:
https://cloud.google.com/storage/docs/uploading-objects

These two lines are suggested for "uploading objects" in Java language tab:

InputStream content = new ByteArrayInputStream("Hello, World!".getBytes(UTF_8));
Blob blob = bucket.create(blobName, content, "text/plain");

The bucket object above can be received from storage object, and that's actually the root of the problem

In all tutorials I have seen so far (particularly in this one: https://support.google.com/dfp_premium/answer/1733127?hl=en), the storage belongs to com.google.api.services.storage.Storage instead of com.google.cloud.storage.Storage, and is instantiated as follows:

        httpTransport = GoogleNetHttpTransport.newTrustedTransport();

        GoogleCredential credential = new GoogleCredential.Builder()
            .setTransport(httpTransport)
            .setJsonFactory(JacksonFactory.getDefaultInstance())
            .setServiceAccountId(SERVICE_ACCOUNT_EMAIL)
            .setServiceAccountScopes(
                Collections.singleton(STORAGE_SCOPE))
            .setServiceAccountPrivateKeyFromP12File(p12File).build();

        storage = new com.google.api.services.storage.Storage.Builder(httpTransport,
            JacksonFactory.getDefaultInstance(), credential)
            .setApplicationName(PROJECT_NAME).build();

After creating a storage object in this way, we may obtain a bucket object as follows:

    com.google.api.services.storage.Storage.Buckets.Get object = storage.buckets().get(BUCKET_NAME);        
    com.google.api.services.storage.model.Bucket bucket = object.execute();

But I can't use com.google.api.services.storage.model.Bucket for uploading files because it doesn't have create() method mentioned above. I need to use com.google.cloud.storage.Bucket instead.

I have checked this stackoverflow question too: Upload image to Google Cloud Storage (Java)
and the examples in the provided links indeed use com.google.cloud.storage.Storage and com.google.cloud.storage.Bucket but they never show HOW to pass credentials and instantiate storage object similar to the code above.

Could someone provide a clear and comprehensive Java snippet showing how to upload a file to Google Cloud Storage, including all necessary credential instantiations of storage object?

EDIT. In other words, I need to customize the instantiation of com.google.cloud.storage.Storage like above, instead of using this line: Storage storage = StorageOptions.getDefaultInstance().getService();

EDIT2. Finally I was able to upload the file after retrieving the bucket by the following code now:

GoogleCredentials credentials = GoogleCredentials.fromStream(new FileInputStream(KEY_JSN)).createScoped(Lists.newArrayList("https://www.googleapis.com/auth/devstorage.read_write"));

Storage storage = StorageOptions.newBuilder().setCredentials(credentials).build().getService();

Bucket bucket = storage.get(BUCKET_NAME);

One single subtle point remaining for me, is that the following line:

Page<Bucket> buckets = storage.list();

returns an empty list. Does it make sense to receive an empty bucket list while being able to receive a particular bucket meta by get(bucket_name) ?

Have you checked [this](https://cloud.google.com/docs/authentication/production#obtaining_and_providing_service_account_credentials_manually)? In particular, section 'Passing the path to the service account key in code'. You can authenticate like that in the Client Libraries. — Mangu, Jul 16 '18 at 10:34
@Mangu thanks for feedback, could you re-check the update above? — mangusta, Jul 17 '18 at 00:55
`storage.list()` should return you a `Page` object, which you can iterate [like this](https://googlecloudplatform.github.io/google-cloud-java/google-cloud-clients/apidocs/com/google/cloud/storage/Storage.html#list-com.google.cloud.storage.Storage.BucketListOption...-). Ignore the options in the `list()` method, it's just options to limits the results. — Mangu, Jul 17 '18 at 07:43
@Mangu that's what I'm doing now, I'm calling `storage.list().iterateAll()` then I'm getting `Iterable` from it, then I'm getting `Iterator` but it is empty. (the code in that link is incorrect, `iterateAll()` returns `Iterable`, not `Iterator`) — mangusta, Jul 18 '18 at 00:41
that's strange, there are 2 buckets associated with the project. maybe the project name (the one that the service account and json key have been created for) should be explicitly specified somewhere when credential or storage is created (although, I guess it is already identified by the json key itself) — mangusta, Jul 18 '18 at 01:06

score 8 · Answer 1 · edited Mar 23 '20 at 08:01

I was able to create and upload to a bucket using the com.google.cloud.storage libraries in the following snippet derived from the documentation:

public void createBlobFromByteArray(String blobName, String jsonPath, String bucketName) throws IOException {
  GoogleCredentials credentials = GoogleCredentials.fromStream(new FileInputStream(jsonPath)).createScoped(Lists.newArrayList("https://www.googleapis.com/auth/cloud-platform"));

  Storage storage = StorageOptions.newBuilder().setCredentials(credentials).build().getService();

  Bucket bucket = storage.create(BucketInfo.of(bucketName));

  Blob blob = bucket.create(blobName, "Hello, World!".getBytes(UTF_8), "text/plain");
}

You need to create a JSON service account key in and download it from the Google Cloud console. Then pass it in the GoogleCredentials method to authenticate 'storage'

thanks for feedback, could you re-check the edit2 of my question above? I'm using the suggested way but receive an empty buckets list — mangusta, Jul 17 '18 at 01:28

score 2 · Answer 2 · edited Aug 04 '21 at 09:36

Export the Google credential JSON from command line:

export GOOGLE_APPLICATION_CREDENTIALS='\path\key.json'

Run this code in application:

Storage storage = StorageOptions.getDefaultInstance().getService();
BlobId blobId = BlobId.of(BUCKET_NAME, OBJECT_NAME);
BlobInfo blobInfo = BlobInfo.newBuilder(blobId).setContentType("text/plain").build();
Blob blob = storage.create(blobInfo, "Hello, World!".getBytes(UTF_8));

Kishore Namala · Answer 3 · 2019-02-14T02:37:03.237

Did you ever figure out the cause ? storage.list() returns the list of buckets in the project. It works fine for me.

EDIT#1: Did you trying using Google API Client libraries instead of the Cloud Client libraries ?

Example code below:

     try {
        httpTransport = GoogleNetHttpTransport.newTrustedTransport();

          // Authenticate using Google Application Default Credentials.
          GoogleCredential gCredential = GoogleCredential.getApplicationDefault();
          if (gCredential.createScopedRequired()) {
            List<String> scopes = new ArrayList<>();
            // Set Google Cloud Storage scope to Full Control.
            scopes.add(ComputeScopes.DEVSTORAGE_FULL_CONTROL);
          }         

          // Create Storage object for listing buckets.
          Storage storage =
              new Storage.Builder(httpTransport, JSON_FACTORY, gCredential)
                  .setApplicationName(APPLICATION_NAME)
                  .build();     

        Buckets buckets = storage.buckets().list(projectId).execute(); <-- give your project identifier.
        for(Bucket buckt:buckets.getItems()) {
            System.out.println(buckt.getName());
        }

    } catch (GeneralSecurityException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

If you haven't tried you can give this a try and see if this works. If not, can be something with your buckets. If works, then most possibly we may have a bug in the cloud client libs.

no, I still did not. I can get a specific bucket by `get(bucket_name)` but `list()` is empty. since the project has only 2 buckets with predefined names, the problem is not that serious, but it could be a trouble if the number of buckets and their names were unknown beforehand — mangusta, Feb 11 '19 at 07:05

Upload a file to Google Cloud Storage (Java)

3 Answers3

Linked