0

I am creating an application which can be run locally or on Google Cloud. To set up google cloud logging I've used Google Cloud Logging, made a cloud logger and basically log using the class below

class CloudLogger():

    def __init__(self, instance_id: str = LOGGER_INSTANCE_ID, instance_zone: str = LOGGER_INSTANCE_ZONE) -> None:
        self.instance_id = instance_id
        self.instance_zone = instance_zone
        self.cred = service_account.Credentials.from_service_account_file(CREDENTIAL_FILE)
        self.client = gcp_logging.Client(project = PROJECT, credentials=self.cred)
        self.res = Resource(type="gce_instance", 
                    labels={
                        "instance_id": self.instance_id, 
                        "zone": self.instance_zone
                        })
        self.hdlr = CloudLoggingHandler(self.client, resource = self.res)
        self.logger = logging.getLogger('my_gcp_logger')
        self.hdlr.setFormatter(logging.Formatter('%(message)s'))
        if not self.logger.handlers:
            self.logger.addHandler(self.hdlr)
            self.logger.setLevel(logging.INFO)

    def info(self, log_this):    
        self.logger.setLevel(logging.INFO)
        self.logger.info(log_this)

I want to have this so that if it is running on the cloud, it uses the GCP logger, and if run locally, it uses python logging. I can either pass in as an argument ("Cloud", "Local") or make it intelligent enough to understand on its own. But I want the underlying logic to be the same so that I can log to cloud/local seamlessly. How would I go about doing this?

Wondering if (maybe) theres some way to create a local logger. And have those local logs parsed to GCP if running on the cloud.

DUDANF
  • 2,618
  • 1
  • 12
  • 42
  • By GCP Logger do you mean Stackdriver? What problem are you having? Is the issue that you want to detect that you are running in the cloud? Running in Google Cloud? Or do you want to support both local log files and Stackdriver? Stackdriver logging is available from your desktop, from inside Google Cloud and from almost anywhere. What is your criteria to determine log locally versus to Stackdriver? Edit your question with details. – John Hanley Nov 18 '19 at 15:06
  • @JohnHanley I want to use the python logger for all logging. And if I'm running on the cloud, pass those logs to Stackdriver. So Primary logger: Python, if cloud: Stackdriver. – DUDANF Nov 18 '19 at 15:17
  • @daudnadeem, I know the problem and I'm in discussion with Google about it. @JohnHanley, when you log thing in Cloud Run (or AppEngine) for example, like `logging.debug`, there is no level detected in stackdriver logging, the log are in grey because there are printed in plain text. If you use the Cloud Logging formater, it works well, but in local, your logs are in FluentD format (JSON, hardly readable). If you use Cloud Function, it works well, because Logger is well formatted in the Function Build step. – guillaume blaquiere Nov 18 '19 at 20:00

1 Answers1

2

I coped with this issue and Google is aware of it (and also for Go). My helper do this:

  • I perform a request to metadata server. A get to http://metadata.google.internal/computeMetadata/v1/project/ with this header Metadata-Flavor: Google
    • If I have a 404, I'm in local, and I set up my logger as local
    • If I have a 2XX, I'm on GCP and I set up the logger to use FluentD format (GCP Cloud Logging)

Not perfect, but enough for me!

guillaume blaquiere
  • 66,369
  • 2
  • 47
  • 76
  • Is there any way to automate this process? I recently learnt it's bad practice for my code to know whether it's on GCP or not. Seems like my writing to python logger, the logs get passed to Stackdriver -> Global -> Python directly. I'm trying to figure out a way to write all logs to python. If on gcp, pass to stackdriver. Else keep in python. – DUDANF Nov 25 '19 at 12:20
  • I don't know where you read this, I'm interesting to dig into this topic. From my understanding, you never have to adapt your process according with the underlying infrastructure. However, for technical data, like log format, you can change the format without changing the business logic, which is the case here. – guillaume blaquiere Nov 25 '19 at 14:07
  • I actually attended google cloud next '19, and over there they had an "ask an expert" area. So I asked an expert about my logging issue and he said all you should do is: 1. Install stack-driver-logging on the machine 2. configure it. 3. it will pick up local logs and push them to stackdriver. Also, when you see your logs, do you look at them thru global>python or under the resource name e.g. – DUDANF Nov 25 '19 at 14:09
  • Very interesting feedback. I was also in London last week. I didn't have this feedback from Google when I submitted them my implementation example. I dug into the [Google Cloud Stackdriver Logging library](https://googleapis.dev/python/logging/latest/usage.html) and the code of the [`get_default_handler`](https://googleapis.dev/python/logging/latest/_modules/google/cloud/logging/client.html#Client.get_default_handler) do something close to my proposal. I will loop again with Google to be sure – guillaume blaquiere Nov 25 '19 at 16:52
  • I mean I spoke to one of the experts, so his opinion might not be google's opinion. If you read [John's](https://stackoverflow.com/questions/59034274/logging-on-either-gcp-or-local/59035200?noredirect=1#comment104315319_59035200) answer, I feel like that is the correct way to approach this. But I haven't got it working yet. – DUDANF Nov 25 '19 at 16:53