2

I would like to upload a JSON file/dictionary to Google Cloud Storage but no matter how I try, I'm getting errors. Is this just not possible?

Below is my Python code:

response = requests.get(url, headers=headers)

json_data = response.json()

storage_client = storage.Client()
bucket = storage_client.bucket("syr_test_bucket1")

blob = bucket.blob(json_data)
blob.upload_from_filename(json_data)

Error I'm getting:

Traceback (most recent call last):
  File "/layers/google.python.pip/pip/bin/functions-framework", line 8, in <module>
    sys.exit(_cli())
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/functions_framework/_cli.py", line 37, in _cli
    app = create_app(target, source, signature_type)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/functions_framework/__init__.py", line 237, in create_app
    spec.loader.exec_module(source_module)
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/workspace/main.py", line 16, in <module>
    blob = bucket.blob(json_data)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/bucket.py", line 716, in blob
    return Blob(
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 221, in __init__
    name = _bytes_to_unicode(name)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/_helpers.py", line 389, in _bytes_to_unicode
    raise ValueError("%r could not be converted to unicode" % (value,))

1 Answers1

2

Wrong use of API function:

The function Bucket.blob(blob_name, ...) accepts a blob-name as input - you provide it a JSON string from your response.json() call.

Unless your JSON is 100% identical to a blobname this wont work.

The upload_from_filename function needs a filename - you provide a string that is not a filename:

blob.upload_from_filename(pretty_json) ... seems like this wants a file on your disk ... not a string in memory... (use upload_from_string for that):

blob.upload_from_string(pretty_json)

or create a file:

with open("dummy.json") as f:
    f.write(pretty_json)

blob.upload_from_filename("dummy.json")

References:

Patrick Artner
  • 50,409
  • 9
  • 43
  • 69
  • Thing is the JSON file is not local. And I'm fetching it from an external URL. I've also tried using blob.upload_from_string(pretty_json) and doesn't work. –  Aug 15 '21 at 09:18
  • Sorry, I've updated my code and made sure the error is exactly the same. Wasn't intentional, just have been trying out 100 different ways to do this. –  Aug 15 '21 at 10:09
  • @moe And why do youi thinkt it is a good idea to put a string containing JSON as parameter to the bucket.blob() function that wants a blobname as input parameter? – Patrick Artner Aug 15 '21 at 10:22
  • I've tried giving it some new name for blobname. But still didn't work. I feel like I'm going about this whole thing wrong or it's simply not possible to upload JSON data to gcs. –  Aug 15 '21 at 10:26
  • @moe create the bucket, create a blob, use that blobs name for `blob = bucket.blob("your_name")` - then use `rv = blob.upload_from_string(pretty_json, content_type='application/json')` and inspect rv (`print(rv)`). Have a look here: [uploading-a-json-to-google-cloud-storage-via-python](https://stackoverflow.com/questions/44876235/uploading-a-json-to-google-cloud-storage-via-python) – Patrick Artner Aug 15 '21 at 10:31
  • I just did that and got the same error. I've seen that SO post, they seem to be using a local JSON file. –  Aug 15 '21 at 11:00