How do I get notified when an object is uploaded to my GCS bucket?

Question

I have an app that uploads photos regularly to a GCS bucket. When those photos are uploaded, I need to add thumbnails and do some analysis. How do I set up notifications for the bucket?

Answer 1

The way to do this is to create a Cloud Pub/Sub topic for new objects and to configure your GCS bucket to publish messages to that topic when new objects are created.

First, let's create a bucket PHOTOBUCKET:

$ gsutil mb gs://PHOTOBUCKET

Now, make sure you've activated the Cloud Pub/Sub API.

Next, let's create a Cloud Pub/Sub topic and wire it to our GCS bucket with gsutil:

$ gsutil notification create \
    -t uploadedphotos -f json \
    -e OBJECT_FINALIZE gs://PHOTOBUCKET

The -t specifies the Pub/Sub topic. If the topic doesn't already exist, gsutil will create it for you.

The -e specifies that you're only interested in OBJECT_FINALIZE messages (objects being created). Otherwise you'll get every kind of message in your topic.

The -f specifies that you want the payload of the messages to be the object metadata for the JSON API.

Note that this requires a recent version of gsutil, so be sure to update to the latest version of gcloud, or run gsutil update if you use a standalone gsutil.

Now we have notifications configured and pumping, but we'll want to see them. Let's create a Pub/Sub subscription:

$ gcloud beta pubsub subscriptions create processphotos --topic=uploadedphotos

Now we just need to read these messages. Here's a Python example of doing just that. Here are the relevant bits:

def poll_notifications(subscription_id):
    client = pubsub.Client()
    subscription = pubsub.subscription.Subscription(
        subscription_id, client=client)
    while True:
        pulled = subscription.pull(max_messages=100)
        for ack_id, message in pulled:
            print('Received message {0}:\n{1}'.format(
                message.message_id, summarize(message)))
            subscription.acknowledge([ack_id])

def summarize(message):
    # [START parse_message]
    data = message.data
    attributes = message.attributes

    event_type = attributes['eventType']
    bucket_id = attributes['bucketId']
    object_id = attributes['objectId']
    return "A user uploaded %s, we should do something here." % object_id

Here is some more reading on how this system works:

https://cloud.google.com/storage/docs/reporting-changes https://cloud.google.com/storage/docs/pubsub-notifications

Answer 2

GCP also offers an earlier version of the Pub/Sub cloud storage change notifications called Object Change Notification. This feature will directly POST to your desired endpoint(s) when an object in that bucket changes. Google recommends the Pub/Sub approach.

https://cloud.google.com/storage/docs/object-change-notification

Answer 3

while using this example! keep in mind two things 1) they have upgraded code to python 3.6 pub_v1 this might not be running on python 2.7 2) while calling poll_notifications(projectid,subscriptionname) pass your GCP project id : e.g bold-idad & subscrition name e.g asrtopic