-1

I've a requirement wherein i have to call a python file stored in GCP Storage bucket, from the driver python file.

Here is the code(being run on my local m/c, but eventually will be scheduled using Airflow on GCP):

from google.cloud import storage
import os

jsonkey = 'googl-cloudstorage-key.json'
storage_client = storage.Client.from_service_account_json(jsonkey)

def download_file_from_bucket(blob_name, file_path, bucket_name):
    print(f" download_file_from_bucket : blob_name file_path : {file_path}")
    try:
        bucket = storage_client.get_bucket(bucket_name)
        print(f" bucket name : {bucket.name}")
        blob = bucket.blob(blob_name)

        print(f" blob.name : {blob.name}, length of blob : {blob.download_as_string} ")

        with open(file_path, "w") as f:
            blob.download_blob_to_file(blob, f)


download_file_from_bucket('file1.py', os.path.join(os.getcwd(),'new_file.py'),'kk_bucket_1')

The file gets downloaded into new_file.py, however the file downloaded is blank

Here is the content of the file uploaded on GCP Storage bucket :

name : file1.py

import sys
print("file1.py, customer is ", sys.argv[1])

What do i need to do, to make this work ?

Pls note : I need to call this file from the driver file, which i plan to do using the subprocess method

Here is the code for that :

import subprocess, os

cust = ['cust1', 'cust2']

for c in cust:
    print(" c -> ", c)
    fileToCall = os.path.join(os.getcwd(), 'file1_1.py')
    print(" file to call ", fileToCall)
    subprocess.run(['python', fileToCall ,c])
Karan Alang
  • 869
  • 2
  • 10
  • 35

1 Answers1

1

I was able to get this to work using the following code :

def download_file_from_bucket(blob_name, file_path, bucket_name):
        bucket = storage_client.get_bucket(bucket_name)

        blob = bucket.blob(blob_name)
        blob.download_to_filename(file_path)
        print("after writing to file")
Karan Alang
  • 869
  • 2
  • 10
  • 35