CloudWatchLogs filterLogEvents empty events

Question

I am trying to get CloudWatchLogs filterLogEvents by sending parameters. some times it work and sends me the exact logs in that time, but for the different lambda function it returns empty events with next token.

parameters = {

'logGroupName' : metricFilter.logGroupName, 'filterPattern' : metricFilter.filterPattern ? metricFilter.filterPattern : "", 'startTime' : timestamp - offset, 'endTime' : timestamp };

i am retrying by sending parameters with next token but its still getting empty events. Does anyone has idea about it?

{
    "events": [],
    "searchedLogStreams": [],
    "nextToken": "long text"
}

score 3 · Answer 1 · answered Nov 15 '21 at 19:28

You might have been previously testing with a query from Cloudwatch Log Insights, then via the StartQuery API and noted that you can not currently paginate over the results GetQueryResults.

Looking to overcome the limitation of 10000 records then you might have tried the FilterLogEvents, that supports pagination. If that is the case, please note that the parameters startTime and endTime from FilterLogEvents are specified in miliseconds while the parameters startTime and endTime from StartQuery must be specified in seconds.

So in order to use the same code used to calculate startTime and endTime from startQuery you need to multiply the timestamp value * 1000.

score 0 · Answer 2 · edited Sep 21 '20 at 02:41

I had the same problem when I tried to capture the aws clould watch logs using aws sdk.

I finally solved this issue when I used logStreamNamePrefix, which in my case is the uuid of the device.

So I created the function below and when I run it the first time I do it without the nextToken and the subsequent times I take the token in the request and call it again using the token to continue the search.

loadCloudWatchLogs(nextToken?: string) {
  // Set the region
  AWS.config.update({
    region: 'sa-east-1',
    credentials: {
      accessKeyId: environment.awsAccessKeyId,
      secretAccessKey: environment.awsSecretAccessKey,
    },
  });

  // Create the CloudWatchLogs service object
  const cloudwatchlogs = new AWS.CloudWatchLogs({ apiVersion: '2014-03-28' });

  // Defines the params attributes pattern.
  let params: AWSCloudWatchParams;

  // Check if token was provided to permorm a next query to AWSCloudWatchLogs.
  if (nextToken) {
    params = {
      logGroupName: 'group-name' /* required */,
      startTime: this.startTime,
      endTime: this.endTime,
      logStreamNamePrefix: this.device,
      filterPattern: `{ ($.device="${this.device}") }`,
      nextToken,
    };
  } else {
    params = {
      logGroupName: 'group-name' /* required */,
      startTime: this.startTime,
      endTime: this.endTime,
      logStreamNamePrefix: this.device,
      filterPattern: `{ ($.device="${this.device}") }`,
    };
  }

  // Execute a filter for logs.
  cloudwatchlogs.filterLogEvents(params, (err, data) => {
    if (err) {
      console.log(err, err.stack);
    } else {
      if (data.searchedLogStreams.length > 0) {
        // Chegk if exists more logs for query.
        const loadMoreLogs = !data.searchedLogStreams[
          data.searchedLogStreams.length - 1
        ].searchedCompletely;

        // Update the token for the next query.
        this.nextToken = data.nextToken;

        data.events.forEach(log => {
          // log proccessing...
        });
      }
    }
  });
}

This function I did in an Angular project, but it will work even with javascript vanila making some adjustments.

CloudWatchLogs filterLogEvents empty events

2 Answers2