AWS lamba start_instances gives pending response but console still shows stopped

Question

I have a lambda function that I have setup to start an instance:

import boto3
ec2 = boto3.client('ec2')
response = ec2.start_instances(
    InstanceIds=['i-xxx']
)
print(response)

The response looks good, showing it going to pending from stopped:

START RequestId: 26c0cf5e-6d70-4701-b0bd-68276b06d30d Version: $LATEST
{
    "StartingInstances": [
        {
            "CurrentState": {"Code": 0, "Name": "pending"},
            "InstanceId": "i-xxxxxx",
            "PreviousState": {"Code": 80, "Name": "stopped"},
        }
    ],
    "ResponseMetadata": {
        "RequestId": "fdab5818-0536-457f-a19e-17fea60100f4",
        "HTTPStatusCode": 200,
        "HTTPHeaders": {
            "x-amzn-requestid": "fdab5818-0536-457f-a19e-17fea60100f4",
            "content-type": "text/xml;charset=UTF-8",
            "content-length": "579",
            "date": "Wed, 16 Dec 2020 18:38:57 GMT",
            "server": "AmazonEC2",
        },
        "RetryAttempts": 0,
    },
}
END RequestId: f2ed2be9-e2f2-4beb-a69b-4cddee35bef4
REPORT RequestId: f2ed2be9-e2f2-4beb-a69b-4cddee35bef4  Duration: 1381.48 ms    Billed Duration: 1382 ms    Memory Size: 256 MB Max Memory Used: 97 MB  Init Duration: 688.96 ms

However, when I look at the console, it still shows as stopped and never starts.

It doesnt seem like it failed to start:

                "StateReason": {
                    "Code": "Client.UserInitiatedShutdown",
                    "Message": "Client.UserInitiatedShutdown: User initiated shutdown",
                },

Execution Based Policy:

{
  "Version": "2012-10-17",
  "Id": "default",
  "Statement": [
    {
      "Sid": "xxxxxxx",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "lambda:InvokeFunction",
      "Resource": "xxxxxxFunction",
      "Condition": {
        "StringEquals": {
          "AWS:SourceAccount": "xxxxxxxx"
        }
      }
    }
  ]
}

It seems like it didn't really try to start. I've used this code to start other instances. I'm wondering if it is a permissions issue, but there is no error. The lambda function execution role has EC2fullaccess.

Note, another datapoint. I have tried further code that uses ssm send command to send a command once its running (after manually starting). If I try that while it's running it succeeds.

Please edit your question to show the lambda execution policy and the cloudwatch logs for the lambda function. You might also like to double check the instance ID. — Tim, Dec 16 '20 at 19:32
OK, I think i have added what you requested. I did check the instance ID and it is correct. I would think it would give an error if it were wrong anyways — eng3, Dec 16 '20 at 20:48
That "User initiated shutdown" is probably key. Can you start and stop the instance manually? — Tim, Dec 16 '20 at 22:07
Yes it works manually. I dont think 'user imitated" means much. I shut it down manually about 15min before i tested the lambda function. so I think what it says is accurate. I just dont understand why it says the state is 'pending" eventhough in the console it shows as "stopped" — eng3, Dec 16 '20 at 22:34
May have gone to pending, then stopped. I'd check CloudTrail to see if it gives you any clues. — Tim, Dec 16 '20 at 22:51

score 2 · Accepted Answer · answered Dec 17 '20 at 00:23

Your EBS volume is encrypted using a KMS key, and the IAM role your Lambda function is using does not have permission to decrypt using that KMS key. In this case, the EC2 instance starts, but then immediately stops.

So there are two possible solutions.

Solution 1: Add your IAM role as a "user" of your KMS key. This requires you to edit the permission policy attached to your KMS key, and add your IAM role as a "user" of the key (to encrypt/decrypt).

Solution 2: Add kms:CreateGrant to your Lambda's IAM role. With this on your IAM role, AWS will allow the EC2 instance to start.

Thanks, this turned out to be the issue. I had added it before and it somehow dropped off. — eng3, Dec 17 '20 at 01:25

AWS lamba start_instances gives pending response but console still shows stopped

1 Answers1