Update existing Log Group using CloudFormation

Question

I have a lambda which has a log group, say LG-1, for which retention is set to Never Expire (default). I need to change this Never Expire to 1 month. I am doing this using CloudFormation. As the log group already exists, when I am trying to deploy my lambda again with the changes in template as :

LambdaFunctionLogGroup:
Type: 'AWS::Logs::LogGroup'
DependsOn: MyLambda
Properties:
  RetentionInDays: 30
  LogGroupName: !Join 
    - ''
    - - /aws/lambda/
      - !Ref MyLambda

the update is failing with error :

[LogGroup Name] already exists.

One possible solution is to delete the log group and then again create it with new changes as shown above which works perfectly well.

But I need to do it without deleting the log group as it will result in the deletion of all the previous logs that I have.

Is there any workaround which is possible ?

Answer 1

@ttulka answered:

".. it is impossible to manipulate resources from CF which already exist out of the stack."

But actually the problem is more general than that and applies to resources created inside of the stack. It has to do with AWS CloudFormation resource "Replacement policy". For some resources the way CloudFormation "updates" the resource is to create a new resource, then delete the old resource (this is called the "Replacement" update policy). This means there is a period of time where you've got two resources of the same type with many of the same properties existing at the same time. But if a certain resource property has to be unique, the two resource can't exist at the same time if they have the same value for this property, so ... CloudFormation blows up.

AWS::Logs::LogGroup.LogGroupName property is one such property. AWS::CloudWatch::Alarm.AlarmName is another example.

A work around is to unset the name so that a random name is used, perform an update, then set the name back to it's predictable fixed value and update again.

---

Rant: It's an annoying problem that really shouldn't exist. I.e. AWS CF should be smart enough to not have to use this weird clunky resource replacement implementation. But ... that's AWS CF for you ...

Answer 2

I think it is impossible to manipulate resources from CF which already exist out of the stack.

One workaround would be to change the name of the Lambda like my-lambda-v2 to keep the old log group together with the new one.

After one month you can delete the old one.

Answer 3

Use customresource Backed lambda within your cloudformation template. The custom resource would be triggered automatically the first time and update your retention policy of the existing log group. If you need it you custom resource lambda to be triggered every time, then use a templating engine like jinja2.

import boto3

client = boto3.client('logs')
response = client.put_retention_policy(
    logGroupName='string',
    retentionInDays=123
)

You can basically make your CF template do (almost) anything you want using Custom Resource

More information (Boto3, you can find corresponding SDK for the language you use) - https://boto3.amazonaws.com/v1/documentation/api/1.9.42/reference/services/logs.html#CloudWatchLogs.Client.put_retention_policy

EDIT: Within the CloudFormation Template, it would look something like the following:

  LogRetentionSetFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src
      Handler: set_retention_period.handler
      Role: !GetAtt LambdaRole.Arn
      DeploymentPreference:
        Type: AllAtOnce

  PermissionForLogRetentionSetup:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:invokeFunction
      FunctionName:
        Fn::GetAtt: [ LogRetentionSetFunction, Arn ]
      Principal: lambda.amazonaws.com

  InvokeLambdaFunctionToSetLogRetention:
    DependsOn: [PermissionForLogRetentionSetup]
    Type: Custom::SetLogRetention
    Properties:
      ServiceToken: !GetAtt LogRetentionSetFunction.Arn
      StackName: !Ref AWS::StackName
      AnyVariable: "Choose whatever you want to send"     
      Tags:
        'owner': !Ref owner
        'task': !Ref task

The lambda function would have the code which sets up the log retention as per the code which I already specified before.

For more information, please google "custom resource backed lambda". Also to get you a head start I have added the ink below: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html