How to download and store a file from web in GCP?

Question

I'd like to use GCP to download a file from web and store it in the Google Cloud Storage Bucket.

For example, I need to download this file: https://gist.github.com/alwerner/5429504/archive/be5e2858bb31b37e0ba386ca2ce349bf0c3e20ef.zip

I've tried this code (https://stackoverflow.com/a/70366028):

from google.cloud import storage

def write_file():
    client = storage.Client()
    bucket = client.get_bucket('bucket-name')
    blob = bucket.blob('path/to/new-blob.txt')
    with blob.open(mode='w') as f:
        for line in object: 
            f.write(line)

Unfortunately, I don't know how to use it in case of web file. How to download my file (by cloud function?) and store in the bucket on GCP?

Answer 1

Get the file and write it doesn't work ?

import requests

with requests.get("https://gist.github.com/alwerner/5429504/archive/be5e2858bb31b37e0ba386ca2ce349bf0c3e20ef.zip", stream=True) as response:
    client = storage.Client()
    bucket = client.get_bucket('bucket-name')
    blob = bucket.blob('path/to/new-blob.txt')

    with blob.open(mode='wb') as f:
        for batch in response.iter_content(1024 * 1024 * 24): # 24 Mb batch
            f.write(batch)

Answer 2

You can use urllib or requests library to get the file from the url, then your existing python code to upload to Cloud Storage.

from google.cloud import storage
import urllib.request

project_id = 'your-project'
bucket_name = 'your-bucket'
destination_blob_name = 'upload.zip'
storage_client = storage.Client()

source_file_name = 'https://gist.github.com/alwerner/5429504/archive/be5e2858bb31b37e0ba386ca2ce349bf0c3e20ef.zip'

def upload_blob(bucket_name, source_file_name, destination_blob_name):
    my_file = urllib.request.urlopen(source_file_name)

    bucket = storage_client.get_bucket(bucket_name)
    blob = bucket.blob(destination_blob_name)

    blob.upload_from_string(my_file.read(), content_type='application/zip')

upload_blob(bucket_name, source_file_name, destination_blob_name)