33

How to fix a drifted AWS CloudFormation stack? I modified a BackendECS Service and it is now "drifted" but there's no information on how to resolve this? This is driving me insane? Zero information on how to resolve this?

Dunedan
  • 7,848
  • 6
  • 42
  • 52
ByteMe
  • 1,159
  • 2
  • 15
  • 28
  • 7
    CloudFormation freaks out whenever there is a small change, and usually can't even detecting it. Advice: use Terraform. I wasted a lot of time on CloudFormation, finally I went back to Terraform. – forzagreen Nov 29 '19 at 13:50
  • I'm honestly baffled. This is a NIGHTMARE to untangle. Like come on AWS... you want me to add "retain" flags, remove resources, and then reimport them? How the hell does that make any sense? Changing resources like that is risky. Why can't I just update the template on record in-place with the drifted values? I haven't even gotten to the production side yet, but this is a minefield. – mukunda Jul 21 '23 at 20:14

2 Answers2

13

Amazon AWS allows troubleshooting of a drifted stack via its "Drift Detection" feature.

Drift detection can be accessed by navigating to CloudFormation > Select Stack > Actions > Detect Drift for current stack

More on this here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/detect-drift-stack.html

Using this diagnostic tool you can view exactly what is different in your AWS configuration as compared to the state that cloudformation expects.

There are a few ways to potentially resolve this.

1) If you did an automated update via an Infrastructure as code service, rollback your change

2) If you manually updated the ECS Service, change the settings back to the expected state as shown in the drift detection. Once your system is back to the expected state your stack will behave normally

3) Delete the current stack resource and re-create it - this is a dangerous way to resolve this as you will lose your update history and rollback states.

sirlanceoflompoc
  • 941
  • 9
  • 11
  • It says my AWS::ECS::Service has been modified, however I'm unable to update the ECS service, it continually searches for a previously deleted ECS service – ByteMe Jan 27 '19 at 08:37
  • I went thru this while trying to sort out if I could recover from someone manually tweaking settings in a S3 bucket and every template edit I made and after "Update Stack" was pressed it just kept saying "no updates performed". Maybe a future release will just let us undo right where the drift diff is? – Mark Jun 29 '19 at 03:11
  • 33
    How can I recreate something that I have deleted? It will have different id. Why they make these useless features? – Matteo Apr 09 '20 at 02:09
  • 1
    There is also forth option, to fix the drift through [importing resources](https://aws.amazon.com/blogs/mt/remediate-drift-via-resource-import-with-aws-cloudformation/). – Marcin Sep 28 '20 at 23:43
  • If delete some resources - ONLY asg can recreate it, other - just all brokes with 'deleted' in drift, without autofiexes. Simper - delete stack and recreate it (when it brokes on delete - we have 'skip broken steps' in menu). Or - very hard - download tempkate, clean it from deleted parts, update, then restore old template. I do it now - 8 hours lost. – dragonfly Nov 08 '20 at 22:14
  • 4
    @Matteo in my case after I manually removed API Gateway, I managed to recreate it by a) removing it from template and updating stack, b) bringing it back exactly as it was before in the template and updating the stack again. – theimowski Dec 29 '21 at 16:38
  • 1
    @theimowski 's solution for recreating a removed resource should be considered part of the answer because it's a relevant scenario. – Bart Swennenhuis Mar 18 '22 at 10:09
8

I had a similar issue with cloudformation and likewise there is still a lack of documentation for how to "undrift" the instances back to what the template specified. I found that commenting out the drifted entity allowed cloudformation to "delete" it and then uncommenting it restores it to the desired state.

That said, I'll second the motion to just use terraform instead as enforcing the template is the default behavior whenever you do an apply.

Kevin Myers
  • 81
  • 1
  • 5