How to convert google cloud pubsub payload to LogEntry object in Golang

Question

I created a Logs Router Sink to export logs to a Pub/Sub. My Golang application is supposed to consume messages from this Pub/Sub via a dedicated Subscription using google client library (Golang).

The messages received on the Subscription are JSON representations of LogEntry objects.

The question: How to unmarshal the JSONs into useful Golang objects?

my first attempt was to unmarshal the JSONs to Entry object. The attempt failed because this object has no JSON mappings for the fields, in particular textPayload JSON field was not unmarshalled
the second attempt was to unmarshal the JSONs to LogEntry object. This object seems to fit the definition, but its main purpose is protobufs (and not JSON). In particular when I tried to unmarshal a LogEntry JSON into it, I got the following error

cannot unmarshal string into Go struct field LogEntry.severity of type ltype.LogSeverity

My 2 cents technique: I print the raw JSON of the log entry. I copy it and paste it in a JSON to Struct website. Copy the struct, tweak it as I wish and use it. Quick, dirty, enough!! Note that you textPayload can't be known by the API, because it's specific to your app logging mechanism — guillaume blaquiere, Aug 02 '21 at 20:06
I think [this](https://cloud.google.com/logging/docs/reference/libraries#logging_list_log_entries-go) is what you're looking for. Let me know. — bernatj, Aug 04 '21 at 13:42
Hello Ivan! It would be fine if you could provide the json you are getting from pub/sub. — Mikhail Andrianov, Aug 08 '21 at 23:53

score 1 · Answer 1 · answered Oct 29 '22 at 03:13

If you get errors about protoPayload or timestamppb.Timestamp it probably is a JSON representation of a protocol buffer. To Marshall/Unmarshal protocol buffers in Go you should use the protojson package. Example:

import (
    ... omitted ...

    _ "google.golang.org/genproto/googleapis/cloud/audit"
    logging "google.golang.org/genproto/googleapis/logging/v2"
    "google.golang.org/protobuf/encoding/protojson"
)

func handlerLogEntry(w http.ResponseWriter, r *http.Request) {
    var le logging.LogEntry
    body, err := ioutil.ReadAll(r.Body)
    if err != nil {
        http.Error(w, "Bad HTTP Request", http.StatusBadRequest)
        log.Printf("handlerLogEntry ReadAll error: %", err)
        return
    }
    if err := protojson.Unmarshal(body, &le); err != nil {
        http.Error(w, "Bad HTTP Request", http.StatusBadRequest)
        log.Printf("handlerLogEntry Unmarshal error: %", err)
        return
    }
    s := fmt.Sprintf("handlerLogEntry: %#v", le)
    log.Printf(s)
    fmt.Fprintln(w, s)
}

This is the best answer. Thank you. It took me hours of experimenting with other structs like the audit LogEntryData from one of the audit packages before finding this post. Just a note that "google.golang.org/genproto/googleapis/logging/v2" is marked deprecated now. The replacement pkg "cloud.google.com/go/logging/apiv2/loggingpb" seems to work fine. — joe miller, Feb 12 '23 at 15:27

score 0 · Answer 2 · answered Aug 09 '21 at 00:01

JSON you are trying to parse contains string instead of an integer. The Log severity code should be one of these:

 type LogSeverity int32

const (
    // (0) The log entry has no assigned severity level.
    LogSeverity_DEFAULT LogSeverity = 0
    // (100) Debug or trace information.
    LogSeverity_DEBUG LogSeverity = 100
    // (200) Routine information, such as ongoing status or performance.
    LogSeverity_INFO LogSeverity = 200
    // (300) Normal but significant events, such as start up, shut down, or
    // a configuration change.
    LogSeverity_NOTICE LogSeverity = 300
    // (400) Warning events might cause problems.
    LogSeverity_WARNING LogSeverity = 400
    // (500) Error events are likely to cause problems.
    LogSeverity_ERROR LogSeverity = 500
    // (600) Critical events cause more severe problems or outages.
    LogSeverity_CRITICAL LogSeverity = 600
    // (700) A person must take an action immediately.
    LogSeverity_ALERT LogSeverity = 700
    // (800) One or more systems are unusable.
    LogSeverity_EMERGENCY LogSeverity = 800
)

Make sure you are setting the correct JSON in Router Sink.

How to convert google cloud pubsub payload to LogEntry object in Golang

2 Answers2