Google Tail log entries: on data event does not receive any message

Question

I need to use Google Logging to get realtime log in my project (using Nodejs/express).

I found the "Tail log entries" is good for me and I want to apply it to my project, but when I try to apply the sample code as in google doc (https://cloud.google.com/logging/docs/samples/logging-tail-log-entries), the event on('data) does not receive any message. I don't think the filter condition matters because I can still use logging.getEntries() in the same condition.

Here is my sample code (It's almost like the sample in google doc):

const express = require("express");
const app = express();
const port = process.env.PORT || 5001;

app.use(express.json());

const server = app.listen(port, () => {
    console.log(`Server is up on port ${port}!`);
});

const {Logging} = require('@google-cloud/logging');
const logging = new Logging();

console.log('running tail log entries test');

const stream = logging
  .tailEntries({
    filter: 'timestamp >= "2021-08-26T15:59:51.000Z"',
  })
  .on('error', console.error)
  .on('data', resp => {
    console.log(resp.entries);
    console.log(resp.suppressionInfo);
    // If you anticipate many results, you can end a stream early to prevent
    // unnecessary processing and API requests.
    // stream.end();
  })
  .on('end', () => {
    console.log('log entry stream has ended');
  });

If anyone has experienced this situation or has related information, please share it to me. Thanks in advance.

UPDATED:

I don't know why but after my post the source code can work correctly. I think there is something wrong with google api and maybe it is fixed.

Answer 1

I am able to repro your issue.

I'm not familiar with Node.JS and find the const stream = .... stream.end() ... code weird but... to your point this is as posted by Google.

const {Logging} = require("@google-cloud/logging");
const logging = new Logging();

const project = process.env.PROJECT;
const logName = `projects/${project}/logs/bunyan_log`;
const log = logging.log(logName);

// Assuming the `logName` need not be duplicated in the `filter`
const filter = `timestamp>="2021-08-26T00:00:00.000Z"`;
console.log(filter);

const stream = log
  .tailEntries({
    filter: "",
  })
  .on('close', () => {console.log("closed");})
  .on('pause', () => {console.log("paused");})
  .on("readbale", () => {console.log("readable");})
  .on('resume', () => {console.log("resumed");})
  .on('error', console.error)
  .on('data', resp => {
    console.log("data");
    console.log(resp.entries);
    console.log(resp.suppressionInfo);
    stream.end();
  })
  .on('end', () => {
    console.log('log entry stream has ended');
  });


const bunyan = require('bunyan');
const {LoggingBunyan} = require("@google-cloud/logging-bunyan");
const loggingBunyan = new LoggingBunyan();

const logger = bunyan.createLogger({
  name: 'test',
  streams: [
    {stream: process.stdout, level: 'info'},
    loggingBunyan.stream('info'),
  ],
});

setInterval(()=>{
  logger.info("Ping");
},1000);

Interestingly, I'm unable to call the underlying REST API directly either:

TOKEN=$(gcloud auth print-access-token)

curl \
--request POST \
--header "Authorization: Bearer ${TOKEN}" \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data "{\"resourceNames\":[\"projects/${PROJECT}\"],\"filter\":\"\"}" \
https://logging.googleapis.com/v2/entries:tail

Yields:

[{
  "error": {
    "code": 400,
    "message": "Invalid value (Object), ",
    "status": "INVALID_ARGUMENT",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.BadRequest",
        "fieldViolations": [
          {
            "description": "Invalid value (Object), "
          }
        ]
      }
    ]
  }
}
]

Although the gcloud command works:

gcloud logging read "logName=\"projects/${PROJECT}/logs/bunyan_log\" timestamp>=\"2021-08-26T00:00:00.000Z\"" \
--project=${PROJECT} \
--format="value(jsonPayload.message)"
Ping
Ping

And adding --log-http suggests it's making a similar call:

=======================
==== request start ====
uri: https://logging.googleapis.com/v2/entries:list?alt=json
method: POST
== headers start ==
b'X-Goog-User-Project': b'[[REDACTED]]'
b'accept': b'application/json'
b'accept-encoding': b'gzip, deflate'
b'authorization': --- Token Redacted ---
b'content-length': b'220'
b'content-type': b'application/json'
b'user-agent': b'[[REDACTED]]'
== headers end ==
== body start ==
{"filter": "logName=\"projects/[[PROJECT]]/logs/bunyan_log\" timestamp>=\"2021-08-26T00:00:00.000Z\"", "orderBy": "timestamp desc", "pageSize": 1000, "resourceNames": ["projects/[[PROJECT]]"]}
== body end ==
==== request end ====
---- response start ----
status: 200
-- headers start --
Cache-Control: private
Content-Encoding: gzip
Content-Type: application/json; charset=UTF-8
Date: Thu, 26 Aug 2021 20:38:36 GMT
Server: ESF
Transfer-Encoding: chunked
Vary: Origin, X-Origin, Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 0
-- headers end --
-- body start --
{
  "entries": [ ... ]
}
-- body end --
total round trip time (request+response): 1.406 secs
---- response end ----
----------------------
Ping
Ping

Do you have a way to output the REST API calls that your code is making?