Error refreshing state: state data in S3 does not have the expected content

Question

When we try to run a terraform script with remote state handling we get the below issue:

Error refreshing state: state data in S3 does not have the expected content. This may be caused by unusually long delays in S3 processing a previous state update. Please wait for a minute or two and try again. If this problem persists, and neither S3 nor DynamoDB are experiencing an outage, you may need to manually verify the remote state and update the Digest value stored in the DynamoDB table to the following value

Answer 1

There are 3 possible workarounds depending on your specific scenario in order to solve it:

Case 1

If you have a backup of your AWS S3 terrform.tfstate file you could restored your state backend "s3" {key = "path/to/terraform.tfstate"} to an older version. Re try terraform init and validate if it works well.

---

Case 2

Remove the out-of-sync entry in AWS DynamoDB Table. There will be a LockID entry in the table containing state and expected checksum which you should delete and that will be re-generated after re-running terraform init.

IMPORTANT CONSIDERATIONS:

• During this process you'll lose the lock state protection, which prevents others from acquiring the lock and potentially 2 people could simultaneously be updating the same resources corrupting your state.
• Please consider using terraform refresh command (https://www.terraform.io/docs/commands/refresh.html), which is used to reconcile the state Terraform knows about (via its state file) with the real-world infrastructure. This can be used to detect any drift from the last-known state, and to update the state file.
• Delete DynamoDB LockID table entry -> Screenshoot:

---

Case 3

If after a terraform destroy you have manually deleted your AWS S3 terraform.tfstate file and then probably trying to spin up a new instance of all the tfstate declared resources, meaning you're working from scratch, you could just update your AWS S3 terrform.tfstate state backend key "s3" {key = "path/to/terraform.tfstate"} to a new one "s3" {key = "new-path/to/terraform.tfstate"}. Retry terraform init and validate that this should works well. This workaround has the limitation that you haven't really solved the root cause, you're just by-passing the problem using a new key for the S3 tfstate.

Answer 2

Encountered the same issue today, but despite Exequiel Barrierero's excellent advice there was no obvious error-- until I found it.

In our case we have an older Terragrunt stack (v0.23.30 / terraform v0.12.31), and one component module for CloudWatch was throwing this same error:

Error: state data in S3 does not have the expected content.

This may be caused by unusually long delays in S3 processing a previous state update. Please wait for a minute or two and try again. If this problem persists, and neither S3 nor DynamoDB are experiencing an outage, you may need to manually verify the remote state and update the Digest value stored in the DynamoDB table to the following value:

... no actual Digest value was supplied. However, I'd recently been refactoring out some of our component modules files to reduce complexity for our stack, and discovered a dangling data.terraform_remote_state for an element that no longer existed -- I'd merged the modules and there was no longer a remote state present for that particular data element.

Once I removed the invalid data.terraform_remote_state reference, both plan and apply completed without any hitches or errors.

Answer 3

This happened to me while I was trying to perform a "terraform init" from a new machine.

So I deleted the DynamoDB lock and tried most of the stuff from this thread, with no luck. An important point while deleting the lock is that I saw multiple records there and remembered we are also using terraform workspaces. So the problem was fixed when I created and switched to the correct workspace in the new machine.

I imagine that the reason behind was that my terraform files had different resources than the state on S3 for the default workspace while running the init command. So I switched to the new resource:

terraform workspace new 'dev'
(or)
terraform workspace select 'dev'

(and then)
terraform init

And after that everything worked out smoothly again.

Answer 4

You can also try to remove -md5 item from dynamodb

Answer 5

I was able to resolve this issue by updating the Digest value on DynamoDB with the one provided by the error:

Error refreshing state: state data in S3 does not have the expected content.

This may be caused by unusually long delays in S3 processing a previous state update. Please wait for a minute or two and try again. If this problem persists, and neither S3 nor DynamoDB are experiencing an outage, you may need to manually verify the remote state and update the Digest value stored in the DynamoDB table to the following value: 8dcbb62b2ddf9b5daebd612fa524a7be

I have looked on the DynamoDB Item that contains the terraform.tfstate-md5 LockID, and replaced the value.

Answer 6

The problem is with the terraform state file. When you use s3 as backend for remote state handling, we get this error because there is a mismatch with the s3 file location and the dynamodb record Try out the below steps.

1) Delete the dyanmodb record for that particular state entry. 2) Delete the state file in s3 location. 3) Now initialize and apply terraform from beginning.

This will create a new entry of state information in dynamodb and add the new state file in s3 and problem will be solved.

Happy coding...

Answer 7

It can happen that there is a mismatch between the Digest field in the dynamodb table and the ETag of the state file on S3. For things to work they should be the same.

Just make sure the Digest on dynamo equals the Etag from the file on S3 by updating the dynamo entry manually.

Answer 8

This happened to me today. Here's my scenario:

• I use terraform workspaces.
• The backend is setup with S3 and dynamoDb.
• For me, the problem occurred on a new machine.

On a new machine, you first have to do terraform init which in my case resulted in the below message

Successfully configured the backend "s3"! Terraform will automatically use this backend unless the backend configuration changes. Error refreshing state: state data in S3 does not have the expected content.

This may be caused by unusually long delays in S3 processing a previous state update. Please wait for a minute or two and try again. If this problem persists, and neither S3 nor DynamoDB are experiencing an outage, you may need to manually verify the remote state and update the Digest value stored in the DynamoDB table to the following value:

In the above message, it is actually suggesting that the digest value in dynamoDB should be blank. When I inspected dynamoDB, there was a digest value against one of the records. I removed it and it started working.

So, in my opinion, here's what happened in my case:

• While experimenting with terraform, I was first trying things without workspaces and also once I introduced workspaces, I had done a few things in default workspace. Although I destroyed this infra but the digest value would have remained somehow in dynamoDB.
• On a new machine, doing terraform init initializes the terraform backend and will also bring in the workspaces that you have. In this case it will be default and any other workspace that you may have created.
• The default workspace is selected by default and since it is expected to be clean, the command failed on finding a digest value in dynamoDB.

Therefore, if you're dealing with the same problem, you can try this:

1. Check if there is a digest value against the lockId that does not have "env:" in it. That is the one that belongs to default.
2. Copy the digest somewhere in case what you're going to do next does not solve the problem and you would need to put it back.
3. Clear the digest value.

Answer 9

It happen because of when we delete the content of s3 bucket and process being processed not completed so for this Please wait for few minutes then try again.