How can I destroy AWS nodegroup when it doesn't appear in CloudFormation and only seems to exist when trying to delete parent cluster or create new?

Question

Cards on the table, I'm a newcomer to AWS so my ignorance is likely the source of my issue.

Sharing the output from my terminal I think is the best way to describe my issue so:

I discovered my issue when I tried to spin up a 4 node AWS EKS cluster via eksctl create cluster --name atp-dev --node-type m5.large --nodes 4

Here is the output from that command:

2021-09-25 10:52:06 [ℹ]  eksctl version 0.49.0
2021-09-25 10:52:06 [ℹ]  using region us-west-2
2021-09-25 10:52:06 [ℹ]  setting availability zones to [us-west-2a us-west-2b us-west-2d]
2021-09-25 10:52:06 [ℹ]  subnets for us-west-2a - public:192.168.0.0/19 private:192.168.96.0/19
2021-09-25 10:52:06 [ℹ]  subnets for us-west-2b - public:192.168.32.0/19 private:192.168.128.0/19
2021-09-25 10:52:06 [ℹ]  subnets for us-west-2d - public:192.168.64.0/19 private:192.168.160.0/19
2021-09-25 10:52:07 [ℹ]  nodegroup "ng-a477426f" will use "ami-0adca766413605f27" [AmazonLinux2/1.19]
2021-09-25 10:52:07 [ℹ]  using Kubernetes version 1.19
2021-09-25 10:52:07 [ℹ]  creating EKS cluster "atp-dev" in "us-west-2" region with un-managed nodes
2021-09-25 10:52:07 [ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup
2021-09-25 10:52:07 [ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-west-2 --cluster=atp-dev'
2021-09-25 10:52:07 [ℹ]  CloudWatch logging will not be enabled for cluster "atp-dev" in "us-west-2"
2021-09-25 10:52:07 [ℹ]  you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR-LOG-TYPES-HERE (e.g. all)} --region=us-west-2 --cluster=atp-dev'
2021-09-25 10:52:07 [ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "atp-dev" in "us-west-2"
2021-09-25 10:52:07 [ℹ]  2 sequential tasks: { create cluster control plane "atp-dev", 3 sequential sub-tasks: { wait for control plane to become ready, create addons, create nodegroup "ng-a477426f" } }
2021-09-25 10:52:07 [ℹ]  building cluster stack "eksctl-atp-dev-cluster"
2021-09-25 10:52:07 [!]  1 error(s) occurred and cluster hasn't been created properly, you may wish to check CloudFormation console
2021-09-25 10:52:07 [ℹ]  to cleanup resources, run 'eksctl delete cluster --region=us-west-2 --name=atp-dev'
2021-09-25 10:52:07 [✖]  creating CloudFormation stack "eksctl-atp-dev-cluster": AlreadyExistsException: Stack [eksctl-atp-dev-cluster] already exists
    status code: 400, request id: 550e6a9d-d919-4a34-a012-7bc362f07c5a
Error: failed to create cluster "atp-dev"

I checked CloudFormation console like it suggests but there are no stacks to interact with I try following directions from the output with eksctl utils describe-stacks --region=us-west-2 --cluster=atp-dev which gives me a whole lot of information.

I can't really parse because this is where my understanding of all this ends:

2021-09-25 10:57:38 [ℹ]  eksctl version 0.49.0
2021-09-25 10:57:38 [ℹ]  using region us-west-2
2021-09-25 10:57:39 [ℹ]  stack/eksctl-atp-dev-nodegroup-ng-7b715a90 = {
  Capabilities: ["CAPABILITY_IAM"],
  CreationTime: 2021-09-22 18:18:00.55 +0000 UTC,
  DeletionTime: 2021-09-23 03:53:22.696 +0000 UTC,
  Description: "EKS nodes (AMI family: AmazonLinux2, SSH access: false, private networking: false) [created and managed by eksctl]",
  DisableRollback: false,
  DriftInformation: {
    StackDriftStatus: "NOT_CHECKED"
  },
  EnableTerminationProtection: false,
  Outputs: [
    {
      OutputKey: "FeaturePrivateNetworking",
      OutputValue: "false"
    },
    {
      ExportName: "eksctl-atp-dev-nodegroup-ng-7b715a90::InstanceRoleARN",
      OutputKey: "InstanceRoleARN",
      OutputValue: "arn:aws:iam::988496401707:role/eksctl-atp-dev-nodegroup-ng-7b715-NodeInstanceRole-TR0AX0LF3N6J"
    },
    {
      OutputKey: "FeatureLocalSecurityGroup",
      OutputValue: "true"
    },
    {
      ExportName: "eksctl-atp-dev-nodegroup-ng-7b715a90::InstanceProfileARN",
      OutputKey: "InstanceProfileARN",
      OutputValue: "arn:aws:iam::988496401707:instance-profile/eksctl-atp-dev-nodegroup-ng-7b715a90-NodeInstanceProfile-ZUE15KQVO72E"
    },
    {
      OutputKey: "FeatureSharedSecurityGroup",
      OutputValue: "true"
    }
  ],
  RollbackConfiguration: {

  },
  StackId: "arn:aws:cloudformation:us-west-2:988496401707:stack/eksctl-atp-dev-nodegroup-ng-7b715a90/6b591dc0-1bd1-11ec-9bec-0a0320ad966b",
  StackName: "eksctl-atp-dev-nodegroup-ng-7b715a90",
  StackStatus: "DELETE_FAILED",
  StackStatusReason: "The following resource(s) failed to delete: [SG]. ",
  Tags: [
    {
      Key: "alpha.eksctl.io/cluster-name",
      Value: "atp-dev"
    },
    {
      Key: "alpha.eksctl.io/nodegroup-name",
      Value: "ng-7b715a90"
    },
    {
      Key: "eksctl.cluster.k8s.io/v1alpha1/cluster-name",
      Value: "atp-dev"
    },
    {
      Key: "alpha.eksctl.io/nodegroup-type",
      Value: "unmanaged"
    },
    {
      Key: "alpha.eksctl.io/eksctl-version",
      Value: "0.49.0"
    },
    {
      Key: "eksctl.io/v1alpha2/nodegroup-name",
      Value: "ng-7b715a90"
    }
  ]
}
2021-09-25 10:57:39 [ℹ]  stack/eksctl-atp-dev-cluster = {
  Capabilities: ["CAPABILITY_IAM"],
  CreationTime: 2021-09-22 18:02:51.122 +0000 UTC,
  Description: "EKS cluster (dedicated VPC: true, dedicated IAM: true) [created and managed by eksctl]",
  DisableRollback: false,
  DriftInformation: {
    StackDriftStatus: "NOT_CHECKED"
  },
  EnableTerminationProtection: false,
  Outputs: [
    {
      ExportName: "eksctl-atp-dev-cluster::SubnetsPrivate",
      OutputKey: "SubnetsPrivate",
      OutputValue: "subnet-0b82f725a2a3635e0,subnet-013021889c8604724,subnet-0ecc53da4fe6b3dde"
    },
    {
      ExportName: "eksctl-atp-dev-cluster::SubnetsPublic",
      OutputKey: "SubnetsPublic",
      OutputValue: "subnet-0f7457b575c99d0c3,subnet-044fa1e27da8b0c7e,subnet-03a4577caf8947eda"
    },
    {
      OutputKey: "FeatureNATMode",
      OutputValue: "Single"
    },
    {
      ExportName: "eksctl-atp-dev-cluster::ServiceRoleARN",
      OutputKey: "ServiceRoleARN",
      OutputValue: "arn:aws:iam::988496401707:role/eksctl-atp-dev-cluster-ServiceRole-S4KL2UIIWWH"
    },
    {
      ExportName: "eksctl-atp-dev-cluster::Endpoint",
      OutputKey: "Endpoint",
      OutputValue: "https://OUTPUTKEY.gr7.us-west-2.eks.amazonaws.com"
    },
    {
      ExportName: "eksctl-ATP-dev-cluster::SharedNodeSecurityGroup",
      OutputKey: "SharedNodeSecurityGroup",
      OutputValue: "[REDACTED]"
    },
    {
      ExportName: "eksctl-ATP-dev-cluster::VPC",
      OutputKey: "VPC",
      OutputValue: "[REDACTED]"
    },
    {
      ExportName: "eksctl-atp-dev-cluster::ClusterSecurityGroupId",
      OutputKey: "ClusterSecurityGroupId",
      OutputValue: "[REDACTED]"
    },
    {
      OutputKey: "ClusterStackName",
      OutputValue: "eksctl-atp-dev-cluster"
    },
    {
      OutputKey: "CertificateAuthorityData",
      OutputValue: "[REDACTED]"
    },
    {
      ExportName: "eksctl-atp-dev-cluster::SecurityGroup",
      OutputKey: "SecurityGroup",
      OutputValue: "[REDACTED]"
    },
    {
      ExportName: "eksctl-atp-dev-cluster::ARN",
      OutputKey: "ARN",
      OutputValue: "arn:aws:eks:us-west-2:988496401707:cluster/atp-dev"
    }
  ],
  RollbackConfiguration: {

  },
  StackId: "arn:aws:cloudformation:us-west-2:988496401707:stack/eksctl-atp-dev-cluster/4d4a7bf0-1bcf-11ec-9822-028a7f03527f",
  StackName: "eksctl-atp-dev-cluster",
  StackStatus: "CREATE_COMPLETE",
  StackStatusReason: "Export eksctl-atp-dev-cluster::VPC cannot be deleted as it is in use by eksctl-atp-dev-nodegroup-ng-7b715a90",
  Tags: [{
      Key: "alpha.eksctl.io/cluster-name",
      Value: "atp-dev"
    },{
      Key: "eksctl.cluster.k8s.io/v1alpha1/cluster-name",
      Value: "atp-dev"
    },{
      Key: "alpha.eksctl.io/eksctl-version",
      Value: "0.49.0"
    }]
}

When you checked the CloudFormation console, did you select the correct region? You should select `US West (Oregon) us-west-2` in the top right corner in the AWS console. Afterwards you should be able to see the CloudFormation stack. — Ervin Szilagyi, Sep 25 '21 at 19:38
Thanks for asking, I did make sure I was in the right region. — atp, Sep 25 '21 at 19:48

Yasen · Answer 1 · 2021-09-26T07:17:18.270

There are two great apps for listing and removing almost any AWS resource, including yours case.

awsls is for listing AWS resources
awsrm is for removing AWS resources

`awsls` is for listing `AWS` resources

jckuester/awsls: A list command for AWS resources

awsls supports listing of over 250 types of resources across 100 different AWS services. The goal is to code-generate a list function for every AWS resource that is covered by the Terraform AWS Provider (currently over 500). If you want to contribute, the generator is here.

`awsrm` is for removing `AWS` resources

jckuester/awsrm: A remove command for AWS resources

This command line tool follows the Unix Philosophy of doing only one thing and doing it well:

It simplifies deleting over 250 AWS resource types across multiple accounts and regions.

Like other Unix-like tools, awsrm reveals its full power when combining it via pipes with other tools, such as awsls for listing AWS resources and grep for filtering by resource attributes.

E.g. removing `aws_eks_cluster` with `awsrm`

To remove all aws_eks_clusters from your account, you may want

awsls aws_eks_cluster | awsrm

To remove aws_eks_cluster from specific region:

awsls aws_eks_cluster -r us-west-1 | awsrm -r us-west-1

To remove aws_eks_cluster from specific profile and region:

awsls aws_eks_cluster -p <yourprofile> -r us-west-1 | awsrm -r us-west-1  -p <yourprofile>

How can I destroy AWS nodegroup when it doesn't appear in CloudFormation and only seems to exist when trying to delete parent cluster or create new?

1 Answers1

awsls is for listing AWS resources

awsrm is for removing AWS resources

E.g. removing aws_eks_cluster with awsrm

`awsls` is for listing `AWS` resources

`awsrm` is for removing `AWS` resources

E.g. removing `aws_eks_cluster` with `awsrm`