What is the best way to monitor stacks on AWS?

Question

I am currently working on projects with Laravel/PHP that I manage with AWS.

I deploy several instances thanks to the CloudFormation service. So i have a lot of LogsGroups.

If I have an error in these LogsGroups, I have to manually search for the error logs.

So, my need is :

When I have an error API or PH message in one of my LogsGroups, like this:

[2021-11-24T13:03:48.075879+00:00] technical.ERROR: TYPE: Trzproject\Trzutils\Exceptions\InvalidJWTException   
MESSAGE: Provided JWT has since expired, as defined by the "exp" claim   
FILE: /var/task/vendor/trzproject/trzcore/src/Trzutils/JWT/JWTService.php   LINE: 64   TRACE:  stack trace disabled ____________________________________________________________________

I want to be alerted with a message (slack, mail, whatever) that tells me in which LogGroup is the error.

I can’t create a LogInsight query because I have a lot of clients and, LogInsight does not allow to make query on a very large amount of LogsGroups.

Thank you in advance for your advice.

Edit : something like this ; https://theithollow.com/2017/12/11/use-amazon-cloudwatch-logs-metric-filters-send-alerts/

but without create x alarms for x logGroups

(sorry for my english)

Sorry I cannot understand well what are your points. Are you going to catch any specific error messages in Cloudwatch? Or are you going to catch if someone failed login many times? — shimo, Nov 27 '21 at 23:25
@shimo thank's for you'r message :) My wish: I want to be alerted if one of my CloudFormation stacks set an error message in the LogsGroups of my all lambda. — Mitsukk, Nov 27 '21 at 23:40
How a CloudFormation stack can send any message? When it fails deploments or updates? — Marcin, Nov 28 '21 at 02:19
@Marcin I edit my post, i just want to say : If i have a error api message on one of my LogsGroups, i want to be notified something like this ; https://theithollow.com/2017/12/11/use-amazon-cloudwatch-logs-metric-filters-send-alerts/ but without create x alarms for x logGroups — Mitsukk, Nov 28 '21 at 03:34
@Mitsukk It has nothing to do with CloudFormation, you have to implement a custom solution for that. — Marcin, Nov 28 '21 at 04:56
@Marcin , it's a good solution to do this : https://towardsaws.com/aws-cloudwatch-error-alerts-450e7017198d Cloudwatch Logs -> trigger Lambda when specifics message -> publish error message from cloudwatch to a SNS Topic Thank's in advance — Mitsukk, Nov 28 '21 at 11:42
[This answer](https://stackoverflow.com/questions/60171078/how-to-send-alert-based-on-log-message-on-cloudwatch) describes a very common pattern to do this. Create a metric filter, create a metric alarm, send the alarm to an SNS topic, and then subscribe to the topic by email, push to Slack (see [this application](https://serverlessrepo.aws.amazon.com/applications/us-east-1/641494176294/aws-sns-to-slack-publisher)), or something else. — hephalump, Nov 28 '21 at 23:35
@hephalump thank's for you'r response This solution it's not good :( because i must to create X Alarm for each Metric FIlter (wer can create one Metric Filter for X LogsGroups ?) — Mitsukk, Nov 29 '21 at 02:56
Another pattern is to stream all your logs to [AWS OpenSearch](https://aws.amazon.com/opensearch-service/) and then use the built-in alerting services instead of SNS (or sent the alerts to SNS). Your only options are to either work with an existing pattern to solve your problem, or build a custom solution. — hephalump, Nov 29 '21 at 03:25
@hephalump thank's again, I’ve never used OpenSearch and I don’t know how it works. Finaly, this option doesn't work : https://aws.amazon.com/fr/blogs/mt/get-notified-specific-lambda-function-error-patterns-using-cloudwatch/ ( Cloudwatch Logs -> trigger Lambda when specifics message -> publish error message to a SNS Topic ) because solution need to specify only ONE LogGroups :(. I really don’t know how to fulfill my need — Mitsukk, Nov 29 '21 at 03:50

Mitsukk · Accepted Answer · 2022-03-09T21:53:56.507

I finally did this and it's a solution to my needs

I created a Lambda that is triggered by CloudWatch Logs. I’ve combined the log groups I want to monitor with CloudWatch Logs as well as a pattern to retrieve only certain messages.

serverless.yml

functions:
  persistLogMessage:
    handler: lambda/PersistLogMessage.php
    timeout: 899 # 14min 59s
    events:
      - cloudwatchLog:
          logGroup: 'MyLogsGroup01'
          filter: '?ERROR ?WARN ?5xx'
      - cloudwatchLog:
          logGroup: 'MyLogsGroup02'
          filter: '?ERROR ?WARN ?5xx'
      ...
    layers:
      - arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}:layer:php-73:1
    role: PersistLogMessageRole
    ...
    resources:
    Conditions:
      Resources:
    #
    PersistLogMessageRole:
      Type: AWS::IAM::Role
      Properties:
        RoleName: ${opt:stage}-${opt:client}-PersistLogMessageRole
        AssumeRolePolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Principal:
                Service:
                  - lambda.amazonaws.com
              Action: sts:AssumeRole
        Policies:
          - PolicyName: PersistLogMessagePolicy
            PolicyDocument:
              Version: '2012-10-17'
              Statement:
                - Effect: "Allow"
                  Action:
                    - logs:CreateLogGroup
                    - logs:CreateLogStream
                    - logs:PutLogEvents
                    - logs:PutRetentionPolicy
                    - logs:DescribeLogStreams
                    - logs:DescribeLogGroups
                  Resource: "*"
                - Effect: Allow
                  Action:
                    - sns:Publish
                  Resource:
                    - "arn:aws:sns:eu-west-3:<account_id>:SnsTopic"

My Lambda:

public function __invoke(array $events): void
{
    $data = $events['awslogs']['data'];
    $this->logger->info('events', $events);
    $dataDecoded = base64_decode($data);
    $logMessage = zlib_decode($dataDecoded);
    /** @var stdClass $stdClassLogMessage */
    $stdClassLogMessage = json_decode($logMessage);
    dump($stdClassLogMessage);

    $params = [
        'Message' => $stdClassLogMessage->logEvents[0]->message,
        'TopicArn' => 'arn:aws:sns:eu-west-3:<account_id>:SnsTopic'
    ];

    $this->snsClient->publish($params);
}

What is the best way to monitor stacks on AWS?

1 Answers1