Setting download directory to Cloud Storage in chrome-drive in Cloud Function

Question

I'm trying to create a Cloud Function that access to a website and download CSV file to Cloud Storage.

I managed to access the site using headless-chrominium and chromedriver.
On my local environment I can set up the download directory like below

options.add_experimental_option("prefs", {
    "download.default_directory": download_dir,
    "plugins.always_open_pdf_externally": True
})

where download_dir is like "/usr/USERID/tmp/"

How in Cloud Function could I assigned the value so that it points to the right Cloud Storage?

score 1 · Accepted Answer · answered Oct 07 '21 at 14:24

As I understand, a GCS bucket cannot be mounted as a local drive in runtime environment used for cloud functions.

Thus, you might need to download the source csv file into the cloud function memory and save it, for example, as a file in the "/tmp" directory.

Then, you can upload it from that location into a GCS bucket. A more detailed explanation how to upload - is provided here: Move file from /tmp folder to Google Cloud Storage bucket

Note: cloud functions have some restrictions - i.e. memory and timeout. Make sure that you allocated (during deployment) enough memory and time to process your csv files.

In addition, make sure that a service account, which is used by your cloud function, has relevant IAM roles for the GCS bucket under discussion.

Setting download directory to Cloud Storage in chrome-drive in Cloud Function

1 Answers1