0

I am trying to run a SSM command on more than 50 EC2 instances of my fleet. By using AWS boto3's SSM client, I am running a specific command on my nodes. My code is given below. After running the code, an unexpected error is showing up.

# running ec2 instances
instances = client.describe_instances()
instance_ids = [inst["InstanceId"] for inst in instances] # might contain more than 50 instances

# run command
run_cmd_resp = ssm_client.send_command(
    Targets=[
        {"Key": "InstanceIds", "Values": inst_ids_all},
    ],
    DocumentName="AWS-RunShellScript",
    DocumentVersion="1",
    Parameters={
        "commands": ["#!/bin/bash", "ls -ltrh", "# some commands"]
    }
)

On executing this, getting below error

An error occurred (ValidationException) when calling the SendCommand operation: 1 validation error detected: Value '[...91 instance IDs...]' at 'targets.1.member.values' failed to satisfy constraint: Member must have length less than or equal to 50.

How do I run the SSM command my whole fleet?

Alonso Karlos
  • 79
  • 4

2 Answers2

1

As shown in the error message and boto3 documentation (link), the number of instances in one send_command call is limited up to 50. To run the SSM command for all instances, splitting the original list into 50 each could be a solution.

FYI: If your account has a fair amount of instances, describe_instances() can't retrieve all instance info in one api call, so it would be better to check whether NextToken is in response.
ref: How do you use "NextToken" in AWS API calls

# running ec2 instances
instances = client.describe_instances()
instance_ids = [inst["InstanceId"] for inst in instances]
while "NextToken" in instances:
    instances = client.describe_instances(NextToken=instances["NextToken"])
    instance_ids += [inst["InstanceId"] for inst in instances]

# run command
for i in range(0, len(instance_ids), 50):
    target_instances = instance_ids[i : i + 50]
    run_cmd_resp = ssm_client.send_command(
        Targets=[
            {"Key": "InstanceIds", "Values": inst_ids_all},
        ],
        DocumentName="AWS-RunShellScript",
        DocumentVersion="1",
        Parameters={
            "commands": ["#!/bin/bash", "ls -ltrh", "# some commands"]
        }
    )
Rohan Kishibe
  • 617
  • 2
  • 10
  • 25
  • Correct, I referred the documentation as well and found the fact that we can only pass up to 50 instances. I was just wondering if there is any other way to pass more than 50 instance IDs since the documentation also mentions that SSM SendCommand can process up to thousand instances. – Alonso Karlos Feb 07 '23 at 05:06
  • 1
    Assign a specific tag for whole instances, then assigning the tag value for `Targets` parameter instead of instance id could work. [link](https://docs.aws.amazon.com/systems-manager/latest/userguide/send-commands-multiple.html) – Rohan Kishibe Feb 07 '23 at 11:33
  • 1
    a good option and already considered. But not possible in my case. Thanks – Alonso Karlos Feb 08 '23 at 08:01
0

Finally after @Rohan Kishibe's answer, I tried to implement below batched execution for the SSM runShellScript.

import math
ec2_ids_all = [...] # all instance IDs fetched by pagination.
PG_START, PG_STOP = 0, 50
PG_SIZE = 50
PG_COUNT = math.ceil(len(ec2_ids_all) / PG_SIZE)

for page in range(PG_COUNT):
    cmd = ssm.send_command(
        Targets=[{"Key": "InstanceIds", "Values": ec2_ids_all[PG_START:PG_STOP]}],
        DocumentVersion="AWS-RunShellScript",
        Parameters={"commands": ["ls -ltrh", "# other commands"]}
    }
    PG_START += PG_SIZE
    PG_STOP += PG_SIZE

In above way, the total number of instance IDs will be distributed in batches and then executed accordingly. One can also save the Command IDs and batch instance IDs in a mapping for future usage.

Alonso Karlos
  • 79
  • 4