Logging in GCP and locally

Question

I am building a system that is intended to run on Virtual Machines in Google Cloud Platform. However, as a form of backup, it may be run locally as well. That being said, my issue currently is with logging. I have two loggers, both work, a local logger and a cloud logger.

Cloud logger

import google.cloud.logging
from google.cloud.logging.handlers import CloudLoggingHandler
from google.oauth2 import service_account

CREDS = google.cloud.logging.Client(
        project=PROJECT, credentials=service_account.Credentials.from_service_account_file(CREDENTIAL_FILE))

class GoogleLogger(CloudLoggingHandler):

    def __init__(self, client=CREDS):
        super(GoogleLogger, self).__init__(client)

def setup_logging():
    """
    This function can be invoked in order to setup logging based on a yaml config in the 
    root dir of this project
    """
    try:
        # with open('logging.yaml', 'rt') as f:
        with open(LOGGING_CONFIG, 'rt') as f:
            config = yaml.safe_load(f.read())
            f.close()
        logging.config.dictConfig(config)
    except Exception:
        print('Error in Logging Configuration. Using default configs')
        print(traceback.format_exc())
        logging.basicConfig(level=logging.INFO)

logging.yaml

version: 1

formatters:
    simple:
        format: "%(name)s - %(lineno)d -  %(message)s"

    complex:
        format: "%(asctime)s - %(name)s | %(levelname)s | %(module)s : [%(filename)s: %(lineno)d] - %(message)s"

    json:
        class: logger.JsonFormatter

handlers:
    console:
        class: logging.StreamHandler
        level: DEBUG
        formatter: complex

    cloud:
        class: logger.GoogleLogger
        formatter: json
        level: INFO

loggers:

    cloud:
        level: INFO
        handlers: [console,cloud]
        propagate: yes

    __main__:
        level: DEBUG
        handlers: [console]
        propagate: yes

I use setup_logging() to set everything up like so:

setup_logging()
logger = logging.getLogger(<type_of_logger>)

can be "cloud" or "__main__"

"main" only logs locally, "cloud" logs both to GCP Stackdriver Logs and locally.

Now if I'm not running on GCP, an error gets thrown here:

CREDS = google.cloud.logging.Client(
        project=PROJECT, credentials=service_account.Credentials.from_service_account_file(CREDENTIAL_FILE))

What is the best way around this? The class GoogleLogger(CloudLoggingHandler): always gets run, and if isn't in GCP it breaks.

An idea is to wrap the class in a try/except block, but that sounds like a horrible idea. How do I make my code smart enough to choose which logger automatically? And if running locally, completely ignore the GoogleLogger?

Edit (Traceback)

File "import_test.py", line 2, in <module>
    from logger import setup_logging
  File "/Users/daudn/Documents/clean_space/tgs_workflow/utils/logger.py", line 16, in <module>
    class GoogleLogger(CloudLoggingHandler):
  File "/Users/daudn/Documents/clean_space/tgs_workflow/utils/logger.py", line 23, in GoogleLogger
    project=PROJECT, credentials=service_account.Credentials.from_service_account_file(CREDENTIAL_FILE))
  File "/usr/local/lib/python3.7/site-packages/google/cloud/logging/client.py", line 123, in __init__
    self._connection = Connection(self, client_info=client_info)
  File "/usr/local/lib/python3.7/site-packages/google/cloud/logging/_http.py", line 39, in __init__
    super(Connection, self).__init__(client, client_info)
TypeError: __init__() takes 2 positional arguments but 3 were given

1) If you use ADC (Application Default Credentials), Google Cloud client libraries will automatically choose which credentials to use. 2) One method to detect if you are running on a Google compute service is to try to read from the Google Metadata server. 3) Combine those methods to run code inside and outside Google Cloud. — John Hanley, Dec 31 '19 at 18:14
@JohnHanley So when my logger is run, everything in my `logger.py` will be run. And that means it will try to instantiate `class GoogleLogger` and if it is not on the cloud it will throw an error and break. I can find out whether I'm local or on GCP, but how do I make my code skip instantiating this class if I am running locally? — DUDANF, Jan 03 '20 at 09:49
Redesign your class. For example, don't do this `class GoogleLogger(CloudLoggingHandler)` Your class should not do anything until you call a function to setup the correct logging. — John Hanley, Jan 03 '20 at 14:38

Danil Smirnov · Answer 1 · 2021-12-16T08:28:56.773

0

This is due to bad inheritance.

Try to pass client into parent's __init__ instead

class LoggingHandlerInherited(CloudLoggingHandler):
    def __init__(self):
        super().__init__(client=google.cloud.logging.Client())

Make sure that you have GOOGLE_APPLICATION_CREDENTIALS in your environments

edited Dec 16 '21 at 08:28

answered Dec 15 '21 at 17:23

Danil Smirnov

1
2

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Dec 15 '21 at 17:45

Logging in GCP and locally

Cloud logger

logging.yaml

Edit (Traceback)

1 Answers1

Linked