3

I am writing a new cloud function and am using the new Google Cloud Logging library as announced at https://cloud.google.com/blog/products/devops-sre/google-cloud-logging-python-client-library-v3-0-0-release.

I am also using functions-framework to debug my code locally before pushing it to GCP. Setup and Invoke Cloud Functions using Python has been particularly useful here.

The problem I have is that when using these two things together I cannot see logging output in my IDE, I can only see print statements. Here's a sample of my code:

from flask import Request
from google.cloud import bigquery
from datetime import datetime
import google.cloud.logging
import logging

log_client = google.cloud.logging.Client()
log_client.setup_logging()

def main(request) -> str:
    #
    # do stuff to setup a bigquery job
    #
    bq_client = bigquery.Client()

    job_config = bigquery.QueryJobConfig(labels={"key": "value"})
    nowstr = datetime.now().strftime("%Y%m%d%H%M%S%f")
    job_id = f"qwerty-{nowstr}"

    query_job = bq_client.query(
        query=export_script, job_config=job_config, job_id=job_id
    )
    print("Started job: {}".format(query_job.job_id))
    query_job.result()  # Waits for job to complete.
    logging.info(f"job_id={query_job.job_id}")
    logging.info(f"total_bytes_billed={query_job.total_bytes_billed}")

    return f"{query_job.job_id} {query_job.state} {query_job.error_result}"

However when I run the function using cloud functions the only output I see is in my terminal is

Started job: qwerty-20220306181905424093

As you can see the call to print(...) has outputted to my terminal but the call to logging.info(...) has not. Is there a way to redirect logging output to my terminal when running locally using functions-framework but not affect logging when the function is running as an actual cloud function in GCP?

jamiet
  • 10,501
  • 14
  • 80
  • 159
  • The first thing I would try is to make the two lines that set up logging for google conditional on if youre running in the cloud or not. Try commenting out those two lines of code and seeing what you get. If you still don't see logging to your console, then work to figure out what you have to do to get basic out-of-the-box Python logging to do what you want it to do. Once you've got local logging working, THEN you can figure out how to have your code do the right thing in both environments. – CryptoFool Mar 06 '22 at 18:29
  • 1
    It is possible that the google library itself supports enabling logging to the console when running locally, but my approach would be to make sure that basic logging works before adding in the google library for either case. – CryptoFool Mar 06 '22 at 18:31
  • That was very good advice. Turns out it was nothing to do with `google.cloud.logging`, it was because my default log level was WARNING. Once I changed that I started seeing logging output in the terminal. I wouldn't have found that without your suggestion, thank you very much. – jamiet Mar 06 '22 at 21:00

1 Answers1

1

Thanks to the advice from @cryptofool I figured out that I needed to change the default logging level to get output to appear in the terminal.

from flask import Request
from google.cloud import bigquery
from datetime import datetime
import google.cloud.logging
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def main(request) -> str:
    #
    # do stuff to setup a bigquery job
    #
    bq_client = bigquery.Client()

    job_config = bigquery.QueryJobConfig(labels={"key": "value"})
    nowstr = datetime.now().strftime("%Y%m%d%H%M%S%f")
    job_id = f"qwerty-{nowstr}"

    query_job = bq_client.query(
        query=export_script, job_config=job_config, job_id=job_id
    )
    print("Started job: {}".format(query_job.job_id))
    query_job.result()  # Waits for job to complete.
    logging.info(f"job_id={query_job.job_id}")
    logging.info(f"total_bytes_billed={query_job.total_bytes_billed}")

    return f"{query_job.job_id} {query_job.state} {query_job.error_result}"

Started job: qwerty-20220306211233889260
INFO:root:job_id=qwerty-20220306211233889260
INFO:root:total_bytes_billed=31457280

However, I still can't any output in the terminal when using google.cloud.logging

from flask import Request
from google.cloud import bigquery
from datetime import datetime
import google.cloud.logging
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)
log_client = google.cloud.logging.Client()
log_client.setup_logging()

def main(request) -> str:
    #
    # do stuff to setup a bigquery job
    #
    bq_client = bigquery.Client()

    job_config = bigquery.QueryJobConfig(labels={"key": "value"})
    nowstr = datetime.now().strftime("%Y%m%d%H%M%S%f")
    job_id = f"qwerty-{nowstr}"

    query_job = bq_client.query(
        query=export_script, job_config=job_config, job_id=job_id
    )
    print("Started job: {}".format(query_job.job_id))
    query_job.result()  # Waits for job to complete.
    logging.info(f"job_id={query_job.job_id}")
    logging.info(f"total_bytes_billed={query_job.total_bytes_billed}")

    return f"{query_job.job_id} {query_job.state} {query_job.error_result}"

Started job: qwerty-20220306211718088936

I think I'll start another thread about this.

jamiet
  • 10,501
  • 14
  • 80
  • 159
  • raised another thread here https://stackoverflow.com/questions/71374317/can-i-see-log-output-in-the-terminal-when-using-functions-framework-and-google-c – jamiet Mar 06 '22 at 21:27