TypeError writing a structured log to google cloud logging from a cloud function

Question

I am trying to write a log from a cloud function into cloud logging, but I am having trouble doing this. My code is the following, where the message is a log like this: "20210319-11:39:58-CET - INFO - this is an info message" and the atttributes are simply strings.

import base64
import json
import os
import pytz
from datetime import datetime
from google.cloud import logging
from google.cloud.logging_v2.resource import Resource

RESOURCE_TYPE = "cloudiot_devices"
PROJECT = os.environ.get("GCP_PROJECT")
DEVICE_REGISTRY = REGISTRY_ID
LOCATION = LOCATION
DATEFORMAT = "%Y%m%d-%H:%M:%S"

def iot_logger(event, context):
    """Triggered from a message on a Cloud Pub/Sub topic.
    Args:
         event (dict): Event payload.
         context (google.cloud.functions.Context): Metadata for the event.
    """
    log = base64.b64decode(event['data']).decode('utf-8')
    attributes = event['attributes']

    print(log)
    print(json.dumps(attributes))

    print("preparing timestamp")
    # Extract elements from log
    date_str,severity,message = log.split(" - ")

    # This is the main content for the log
    log_content = {"message":message}

    # turn log timestamp str to an aware datetime object for our timestamp
    tzname = date_str.split("-")[-1]
    # remove time zone to avoid tripping up the datetime module
    date_str = date_str[:-4]
    date_unaware = datetime.strptime(date_str,DATEFORMAT)
    date_aware = date_unaware.astimezone(pytz.timezone(tzname))

    print(f"done preparing timestamp: {date_aware}")

    # extract attributes
    logger_name = attributes["logger_name"]
    device_num_id = attributes["device_num_id"]

    print("creating resource")
    # create resource 
    resource_labels = {
        'project_id':PROJECT,
        'device_num_id':device_num_id,
        'device_registry_id':DEVICE_REGISTRY,
        'location':LOCATION
    }
    log_resource = Resource(type=RESOURCE_TYPE,labels=resource_labels)

    print("initiating client")
    # initiate logging client
    log_client = logging.Client()

    logger = log_client.logger(logger_name)

    print("creating structured log")
    logger.log_struct(log_content,severity=severity,resource=log_resource,timestamp=date_aware)

However, I keep facing issues with the log_struct command and I haven't been able to figure out what is the problem. Initially I was passing a string to the timestamp and it wouldn't work, so now I am passing a date object. But now the error is different: EDIT: There's more than one:

Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/protobuf/json_format.py", line 589, in _ConvertFieldValuePair
    self.ConvertMessage(value, sub_message)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/protobuf/json_format.py", line 485, in ConvertMessage
    self._ConvertFieldValuePair(value, message)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/protobuf/json_format.py", line 603, in _ConvertFieldValuePair
    raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
google.protobuf.json_format.ParseError: Failed to parse labels field: expected string or bytes-like object.

Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/protobuf/json_format.py", line 589, in _ConvertFieldValuePair
    self.ConvertMessage(value, sub_message)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/protobuf/json_format.py", line 485, in ConvertMessage
    self._ConvertFieldValuePair(value, message)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/protobuf/json_format.py", line 603, in _ConvertFieldValuePair
    raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
google.protobuf.json_format.ParseError: Failed to parse labels field: expected string or bytes-like object.

"Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/functions_framework/__init__.py", line 149, in view_func
    function(data, context)
  File "/workspace/main.py", line 64, in iot_logger
    logger.log_struct(log_content,severity=severity,resource=log_resource,timestamp=date_aware)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/logging_v2/logger.py", line 180, in log_struct
    self._do_log(client, StructEntry, info, **kw)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/logging_v2/logger.py", line 133, in _do_log
    client.logging_api.write_entries([api_repr])
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/logging_v2/_gapic.py", line 140, in write_entries
    log_entry_pbs = [_log_entry_mapping_to_pb(entry) for entry in entries]
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/logging_v2/_gapic.py", line 140, in <listcomp>
    log_entry_pbs = [_log_entry_mapping_to_pb(entry) for entry in entries]
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/logging_v2/_gapic.py", line 507, in _log_entry_mapping_to_pb
    ParseDict(mapping, entry_pb)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/protobuf/json_format.py", line 454, in ParseDict
    parser.ConvertMessage(js_dict, message)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/protobuf/json_format.py", line 485, in ConvertMessage
    self._ConvertFieldValuePair(value, message)
  File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/protobuf/json_format.py", line 597, in _ConvertFieldValuePair
    raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
google.protobuf.json_format.ParseError: Failed to parse resource field: Failed to parse labels field: expected string or bytes-like object.."

It says it needs a string or bytes-like object, but I am not sure where. I imagine it's not related to the date, as it was giving me an error when it was a string. Also the resource I believe should work, I am looking at a very similar example on this stackoverflow question, but it doesn't work for me:

Google Cloud Functions Python Logging issue

What am I doing wrong?

@jabbson, there's more than one actually but I thought the main one would be the actual error. I am editing the question to include a couple more logs that appeared when the function crashed. — leonardo, Mar 19 '21 at 13:42
Ok, so it actually happens in `logger.log_struct` statement, specifically for log_resource, could you print it out - both type and labels? — jabbson, Mar 19 '21 at 14:01
@jabbson resource type is the string "cloudiot_devices" and the contents in labels are: 'project_id': is the name of the project from the environment variable, 'device_num_id' is a string with a number like "2498027039031154",'device_registry_id' is a string with the name of the iot core registry,'location' is "europe-west-1". I have used the resource to upload logs in the past through the logging library directly from the iot device, but it wold lag behind on uploading logs so now I am doing it through this pub/sub triggered function. — leonardo, Mar 19 '21 at 14:26
@jabbson I am not sure actually, since the cloud function manages this itself. I can choose the python 3.8 environment and added google-cloud-logging on requirements.txt but I do not know the version. I assume it loads the most recent version when it deploys the function. — leonardo, Mar 19 '21 at 16:06
just tried to create a cloud function with the [code very similar to yours](https://pastebin.com/raw/zEYTnbyX) and two packages in requirements.txt - `pytz` and `google-cloud-logging`. Works like a charm. What do you think could be different about yours? — jabbson, Mar 19 '21 at 22:31
@jabbson So after your comment I had a closer look into it because as far as I could tell the code was right and your confirmation that on your end it worked just made it more weird. I got it to work now and it could've been one of two things. On the code I shared, I had as resource type "cloudiot_devices", so I looked up the docs and saw the "s" was a typo. Another change I made was hard code my project id, as I found that the environment variable was not getting picked up correctly. Not sure if one or both were the cause, but now it works. I feel silly now. Thanks so much for the help! — leonardo, Mar 22 '21 at 10:10

score 0 · Answer 1 · answered Mar 22 '21 at 13:09

I solved it with the help of @jabbson. I ran into a few issues, but mostly I was giving it a couple of wrong things without realizing it.

I was using the environment variables as pointed out here: https://cloud.google.com/functions/docs/env-var#nodejs_8_python_37_and_go_111 but for some reason I was getting null for project id. Also I had a typo in my resource_type.

Lastly, I had to play around with the timestamp until it actually worked like I wanted it to. The LogEntry documentation says it's a string, but in this case I had to pass it a datetime object and converting local time to UTC so I could actually find it in cloud-logging.

TypeError writing a structured log to google cloud logging from a cloud function

1 Answers1