Get Cloud Storage upload response

Question

I am uploading a file to a Cloud Storage bucket using the Python SDK:

from google.cloud import storage

bucket = storage.Client().get_bucket('mybucket')
df = # pandas df to save
csv = df.to_csv(index=False)
output = 'test.csv'
blob = bucket.blob(output)
blob.upload_from_string(csv)

How can I get the response to know if the file was uploaded successfully? I need to log the response to notify the user about the operation.

I tried with:

response = blob.upload_from_string(csv)

but it always return a None object even when the operation has succeded.

if you are using REST, just step a if loop to see if the http return code == 200. — mac_online, Sep 30 '21 at 09:12
I was trying to reproduce your code, but I don't quite understand which type of response you are expecting, do you want as the response the object created or another message just alerting the user the object was created successfully? — Vicky, Oct 01 '21 at 13:17
@Vicky Just a message alerting me whether the object was created successfully or not — Luiscri, Oct 01 '21 at 14:08

score 0 · Answer 1 · answered Sep 30 '21 at 09:34

You can try with tqdm library.

import os
from google.cloud import storage
from tqdm import tqdm

def upload_function(client, bucket_name, source, dest, content_type=None):
  bucket = client.bucket(bucket_name)
  blob = bucket.blob(dest)
  with open(source, "rb") as in_file:
    total_bytes = os.fstat(in_file.fileno()).st_size
    with tqdm.wrapattr(in_file, "read", total=total_bytes, miniters=1, desc="upload to %s" % bucket_name) as file_obj:
      blob.upload_from_file(file_obj,content_type=content_type,size=total_bytes,
      )
      return blob

if __name__ == "__main__":
  upload_function(storage.Client(), "bucket", "C:\files\", "Cloud:\blob.txt", "text/plain")

Vicky · Answer 2 · 2021-10-06T10:27:24.730

Regarding how to get notifications about changes made into the buckets there is a few ways that you could also try:

Using Pub/Sub - This is the recommended way where Pub/Sub notifications send information about changes to objects in your buckets to Pub/Sub, where the information is added to a Pub/Sub topic of your choice in the form of messages. Here you will find an example using python, as in your case, and using other ways as gsutil, other supported languages or REST APIs.
Object change notification with Watchbucket: This will create a notification channel that sends notification events to the given application URL for the given bucket using a gsutil command.
Cloud Functions with Google Cloud Storage Triggers using event-driven functions to handle events from Google Cloud Storage configuring these notifications to trigger in response to various events inside a bucket—object creation, deletion, archiving and metadata updates. Here there is some documentation on how to implement it.
Another way is using Eventarc to build an event-driven architectures, it offers a standardized solution to manage the flow of state changes, called events, between decoupled microservices. Eventarc routes these events to Cloud Run while managing delivery, security, authorization, observability, and error-handling for you. Here there is a guide on how to implement it.

Here you’ll be able to find related post with the same issue and answers:

Using Storage-triggered Cloud Function.
With Object Change Notification and Cloud Pub/Sub Notifications for Cloud Storage.
Answer with a Cloud Pub/Sub topic example.

lulu · Answer 3 · 2023-02-01T16:26:34.993

You can verify if the upload gets any error, then use the exception's response methods:

def upload(blob,content):
    try:
        blob.upload_from_string(content)
    except Exception as e:
        status_code = e.response.status_code
        status_desc = e.response.json()['error']['message']
    else:
        status_code = 200
        status_desc = 'success' 
    finally:
        return status_code,status_desc

Refs:

Get Cloud Storage upload response

3 Answers3