0

I have a Python code cloud function which gets triggered as soon as particular csv file comes in to a bucket. It reads that csv file & transforms the records in to JSON format & then publish it to the topic. below is the code:-

from google.cloud import storage

client = storage.Client()
import csv
import json
import datetime
import time


def test_multi_topic(event, context):
    """Triggered by a change to a Cloud Storage bucket.
    Args:
         event (dict): Event payload.
         context (google.cloud.functions.Context): Metadata for the event.
    """
    file = event
    print(f"Processing file: {file['name']}.")
    print(f"file :{file}.")

    bucket_name = 'ingka-cbdexpose-iipb'
    bucket = client.get_bucket(bucket_name)
    blob = bucket.get_blob('cbd/sync/cbd-data/test_file.csv')
    dest_file = '/tmp/test_file.csv'
    blob.download_to_filename(dest_file)
    processedfilename = 'cbd/sync/processed/test_file' + str(datetime.datetime.now()) + '.csv'
    blobprocessed = bucket.blob(processedfilename)
    import pandas as pd
    df = pd.read_csv(dest_file, encoding='utf-8', dtype=str)
    df = df.assign(FILE_TYPE='TESTFL')
    data = df.to_json(orient="records", lines=True).split('\n')
    # jsonValue = data
    from google.cloud import pubsub_v1
    publisher = pubsub_v1.PublisherClient()
    topic_1 = publisher.topic_path('ingka-cbd-exposecbd-devp', 'test-dev')
    topic_2 = publisher.topic_path('ingka-cbd-exposecbd-devp', 'test2-dev')
    for data in data:
        data = data.encode("utf-8")
        response_1 = publisher.publish(topic_1, data=data)
   
        response_2 = publisher.publish(topic_2, data=data)
 
    blob.delete()
    try:
        with open(dest_file, "rb") as my_file:
            blobprocessed.upload_from_file(my_file)
    except:
        print("An exception occurred")

Now this code runs successful & my files does get published to the prescribed topic. But, it always logs an error that -"'NoneType' object has no attribute 'download_to_filename'". I found one stackflow answer on this link {https://stackoverflow.com/questions/51439904/attributeerror-nonetype-object-has-no-attribute-upload-from-filename}. when I implement this, I start getting different error as :- google.resumable_media.common.InvalidResponse: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>). Can anyone please help me in resolving the issue....

SUNDARAM SRIVASTAVA
  • 140
  • 2
  • 13
  • Can you confirm there isn't a path error? If you're getting a NoneType error, it means the blob holds no value because 'cbd/sync/cbd-data/test_file.csv' was not found. A 404 error is showing that link it's looking for doesn't exist. – PhD Rookie Dec 18 '20 at 17:25
  • @PhDRookie :- There is no Path error. Actually the flow is :- First the file comes at path {cbd/sync/cbd-data/test_file.csv}. This triggers the event of cloud function & it starts processing by creating the temporary file {dest_file = '/tmp/test_file.csv'} at run time . when file gets processed in to processed folder,it deletes the temporary file {/tmp/test_file.csv}.Also the file test_file.csv will be removed from the path {cbd/sync/cbd-data/test_file.csv} on successful processing to topic – SUNDARAM SRIVASTAVA Dec 18 '20 at 17:47
  • It got resolved by using this:- if blob is not None and blob.exists(client): blob.download_to_filename(dest_file) – SUNDARAM SRIVASTAVA Jan 07 '21 at 21:32

0 Answers0