InvalidParameterCombination: "Duplicate tag key found in request" for Amazon Neptune Service CloudFormation

Question

When I deploy my CloudFormation Stack, I receive the following error:

Duplicate tag key found in request: Name (Service: AmazonNeptune; Status Code: 400; Error Code: InvalidParameterCombination; Request ID: ffffc8f8-ac83-4eb0-8794-47c6f5ff5ed1; Proxy: null)

This error only happens when I deploy with multiple stack templates. If I deploy the template that fails (the sub-child stack) on its own, then it will succeed. It only fails when this stack is a child of my other templates.

There are multiple layers. I have the root stack with multiple child stacks(one of which is neptune-application-map.yaml), and then I have a sub-child stack (neptune.yaml which is a NeptuneStack resource). The sub-child stack is the failing one.

Please note, that this error happens for multiple Resources within the failing stack (this sub-child stack). The resources that this error occurs on are:

NeptuneDBClusterParameterGroup
NeptuneDBParameterGroup
NeptuneSubnetGroup

Since there are optional, I tried to remove them, but then got the same error again for NeptuneDBCluster Resource within the same sub-child stack.

Here, I am attaching yaml for reference.

neptune-application-map.yaml

AWSTemplateFormatVersion: '2010-09-09'
Description: Neptune full stack with gremlin and rd4j console
Parameters:
  Environment:
    Description: dev/staging/prod
    Type: String
    AllowedValues: ["dev", "staging", "prod"]
    MaxLength: 15
  
  SubnetIds:
    Type: "List<AWS::EC2::Subnet::Id>"
    Description: Neptune VPC Subnets

  DefaultSecurityGroupId:
    Type: AWS::EC2::SecurityGroup::Id
  
  VpcId:
    Type: AWS::EC2::VPC::Id

  DbInstanceType:
    Description: Neptune DB instance type
    Type: String
    Default: db.r5.large
    AllowedValues:
      - db.t3.medium
      - db.r4.large
      - db.r4.xlarge
      - db.r4.2xlarge
      - db.r4.4xlarge
      - db.r4.8xlarge
      - db.r5.large
      - db.r5.xlarge
      - db.r5.2xlarge
      - db.r5.4xlarge
      - db.r5.8xlarge
      - db.r5.12xlarge
    ConstraintDescription: >-
      Must be a valid Neptune instance type. Note that for Stockholm and OSU
      only R5 and T3 instances are available.

  DBReplicaIdentifierSuffix:
    Description: >-
      OPTIONAL: The ID for the Neptune Replica to use. Empty means no read
      replica.
    Type: String
    Default: ''
  
  DBClusterPort:
    Type: String
    Default: '8182'
    Description: Enter the port of your Neptune cluster
  
  EC2ClientInstanceType:
    Description: EC2 client instance
    Type: String
    Default: r5.2xlarge # t3.medium
    AllowedValues:
      - t3.micro
      - t3.small
      - t3.medium
      - m5.large
      - m5.xlarge
      - m5.2xlarge
      - m5.4xlarge
      - m5.12xlarge
      - r4.large
      - r4.xlarge
      - r4.2xlarge
      - r4.4xlarge
      - r4.8xlarge
      - r5.large
      - r5.xlarge
      - r5.2xlarge
      - r5.4xlarge
      - r5.12xlarge
    ConstraintDescription: >-
      Must be a valid EC2 instance type. Note some regions support limited
      instance types only. Ex: Stockholm and OSU does not support R4 instances
  
  NeptuneQueryTimeout:
    Type: Number
    Default: 20000
    Description: Neptune Query Time out (in milliseconds)
  
  NeptuneEnableAuditLog:
    Type: Number
    Default: 0
    AllowedValues:
      - 0
      - 1
    Description: Enable Audit Log. 0 means disable and 1 means enable.
  
  IamAuthEnabled:
    Type: String
    Default: 'false'
    AllowedValues:
      - 'true'
      - 'false'
    Description: Enable IAM Auth for Neptune.
  
  SetupGremlinConsole:
    Type: String
    Default: 'true'
    AllowedValues:
      - 'true'
      - 'false'
    Description: Setup Gremlin console.
  
  SetupRDF4JConsole:
    Type: String
    Default: 'true'
    AllowedValues:
      - 'true'
      - 'false'
    Description: Setup RDF4J console.
  
  AttachBulkloadIAMRoleToNeptuneCluster:
    Type: String
    Default: 'true'
    AllowedValues:
      - 'true'
      - 'false'
    Description: Attach Bulkload IAM role to cluster
  
  NotebookInstanceType:
    Description: >-
      SageMaker Notebook instance type. Please refer
      https://aws.amazon.com/sagemaker/pricing/ for uptodate allowed instance
      type in aws region and https://aws.amazon.com/neptune/pricing/ for
      pricing.
    Type: String
    Default: none
    AllowedValues:
      - none
      - ml.t2.medium
      - ml.t2.large
      - ml.t2.xlarge
      - ml.t2.2xlarge
      - ml.m4.xlarge
      - ml.m4.2xlarge
      - ml.m4.4xlarge
      - ml.m4.10xlarge
      - ml.m4.16xlarge
      - ml.m5.large
      - ml.m5.xlarge
      - ml.m5.2xlarge
      - ml.m5.4xlarge
      - ml.m5.12xlarge
      - ml.m5.24xlarge
      - ml.m5d.large
      - ml.m5d.xlarge
      - ml.m5d.2xlarge
      - ml.m5d.4xlarge
      - ml.m5d.12xlarge
      - ml.m5d.24xlarge
      - ml.c4.large
      - ml.c4.xlarge
      - ml.c4.2xlarge
      - ml.c4.4xlarge
      - ml.c4.8xlarge
    ConstraintDescription: Must be a valid SageMaker instance type.
  NeptuneSagemakerNotebookStartupScript:
    Description: 'OPTIONAL: Startup script additions for the notebook instance.'
    Type: String
    Default: ''
  S3BucketPrefix:
    Type: String
    Default: 'binxio-public'
  
  CFNCustomProviderZipFileName:
    Type: String
    Default: 'lambdas/cfn-secret-provider-1.3.1.zip'
Rules:
  DBInstanceTypeVsRegion:
    RuleCondition: !Or
      - !Equals
        - !Ref 'AWS::Region'
        - eu-north-1
      - !Equals
        - !Ref 'AWS::Region'
        - us-gov-east-1
      - !Equals
        - !Ref 'AWS::Region'
        - ca-central-1
      - !Equals
        - !Ref 'AWS::Region'
        - me-south-1
      - !Equals
        - !Ref 'AWS::Region'
        - cn-northwest-1
    Assertions:
      - Assert:
          'Fn::Contains':
            - - db.t3.medium
              - db.r5.large
              - db.r5.xlarge
              - db.r5.2xlarge
              - db.r5.4xlarge
              - db.r5.12xlarge
            - !Ref DbInstanceType
        AssertDescription: >-
          Only R5 and T3 Instances are available in eu-north-1 (ARN),
          us-gov-east-1 (OSU), ca-central-1 (YUL), me-south-1 (BAH) and
          cn-northwest-1 (ZHY)
  EC2ClientInstanceTypeVsRegion:
    RuleCondition: !Or
      - !Equals
        - !Ref 'AWS::Region'
        - eu-north-1
      - !Equals
        - !Ref 'AWS::Region'
        - us-gov-east-1
      - !Equals
        - !Ref 'AWS::Region'
        - ca-central-1
      - !Equals
        - !Ref 'AWS::Region'
        - me-south-1
      - !Equals
        - !Ref 'AWS::Region'
        - cn-northwest-1
    Assertions:
      - Assert: !Not
          - 'Fn::Contains':
              - - r4.large
                - r4.xlarge
                - r4.2xlarge
                - r4.4xlarge
                - r4.8xlarge
              - !Ref EC2ClientInstanceType
        AssertDescription: >-
          R4 Instances are not available in eu-north-1 (ARN), us-gov-east-1
          (OSU), ca-central-1 (YUL), me-south-1 (BAH) and cn-northwest-1 (ZHY)
  SagemakerNotebookInstanceTypeVsRegion:
    RuleCondition: !Or
      - !Equals
        - !Ref 'AWS::Region'
        - us-gov-east-1
      - !Equals
        - !Ref 'AWS::Region'
        - us-gov-west-1
      - !Equals
        - !Ref 'AWS::Region'
        - cn-northwest-1
    Assertions:
      - Assert:
          'Fn::Contains':
            - - none
            - !Ref NotebookInstanceType
        AssertDescription: >-
          Neptune Sagemaker notebooks are not available in us-gov-east-1 (OSU),
          us-gov-west-1 (PDT) and cn-northwest-1 (ZHY)
Conditions:
  CreateDBReplicaInstance: !Not
    - !Equals
      - !Ref DBReplicaIdentifierSuffix
      - ''
  AZ3NotPresent: !Equals
    - !Ref 'AWS::Region'
    - ca-central-1
  AZ3Present: !Not
    - !Condition AZ3NotPresent
  CreateSagemakerNotebook: !Equals
    - !Ref NotebookInstanceType
    - none
Resources:
  CFNSecretProvider:
    Type: 'AWS::CloudFormation::Stack'
    Properties:
      TemplateURL: cfn-secret-provider-1.3.1.yaml
      Parameters:
        S3BucketPrefix: !Ref S3BucketPrefix
        CFNCustomProviderZipFileName: !Ref CFNCustomProviderZipFileName
        Environment: !Ref Environment
  PrivateKey:
    Type: Custom::RSAKey
    DependsOn: CFNSecretProvider
    Properties:
      Name: !Sub '/${Environment}-${AWS::Region}/neptune-stack/private-key'
      Version: v1
      NoEcho: False
      ServiceToken: !GetAtt [ CFNSecretProvider, Outputs.LambdaArn ]
  EC2CustomKeyPair:
    Type: Custom::KeyPair
    DependsOn: [ PrivateKey, CFNSecretProvider ]
    Properties:
      Name: !Sub '${Environment}-${AWS::Region}-custom-key-pair'
      NoEcho: False
      PublicKeyMaterial: !GetAtt 'PrivateKey.PublicKey'
      ServiceToken:  !GetAtt [ CFNSecretProvider, Outputs.LambdaArn ]
  NeptuneStack:
    Type: 'AWS::CloudFormation::Stack'
    DependsOn: [ EC2CustomKeyPair, CFNSecretProvider ]
    Properties:
      TemplateURL: neptune.yaml
      TimeoutInMinutes: 60
      Parameters:
        Environment: !Ref Environment
        DBClusterPort: !Ref DBClusterPort
        DbInstanceType: !Ref DbInstanceType
        NeptuneQueryTimeout: !Ref NeptuneQueryTimeout
        NeptuneEnableAuditLog: !Ref NeptuneEnableAuditLog
        IamAuthEnabled: !Ref IamAuthEnabled
        SubnetIds: !Join [ ",", !Ref SubnetIds ]
        DefaultSecurityGroupId: !Ref DefaultSecurityGroupId
        VpcId: !Ref VpcId

  NeptuneEC2Client:
    Type: 'AWS::CloudFormation::Stack'
    DependsOn: EC2CustomKeyPair
    Properties:
      TemplateURL: !Join
        - ''
        - - >-
            https://s3.amazonaws.com/aws-neptune-customer-samples/v2/cloudformation-templates/neptune-ec2-client.json
      TimeoutInMinutes: 30
      Parameters:
        Env: !Ref Environment
        EC2SSHKeyPairName: !Sub '${Environment}-${AWS::Region}-custom-key-pair'
        EC2ClientInstanceType: !Ref EC2ClientInstanceType
        SetupGremlinConsole: !Ref SetupGremlinConsole
        SetupRDF4JConsole: !Ref SetupRDF4JConsole
        AttachBulkloadIAMRoleToNeptuneCluster: !Ref AttachBulkloadIAMRoleToNeptuneCluster
        VPC: !Ref VpcId
        Subnet: !Select [0, !Ref SubnetIds]
        NeptuneDBCluster: !GetAtt
          - NeptuneStack
          - Outputs.DBClusterId
        NeptuneDBClusterEndpoint: !GetAtt
          - NeptuneStack
          - Outputs.DBClusterEndpoint
        NeptuneDBClusterPort: !GetAtt
          - NeptuneStack
          - Outputs.DBClusterPort
        NeptuneLoadFromS3IAMRoleArn: !GetAtt
          - NeptuneStack
          - Outputs.NeptuneLoadFromS3IAMRoleArn
        NeptuneSG: !GetAtt
          - NeptuneStack
          - Outputs.NeptuneSecurityGroup
        NeptuneEC2InstanceProfile: !GetAtt
          - NeptuneStack
          - Outputs.NeptuneEC2InstanceProfile
  NeptuneSagemakerNotebook:
    Type: 'AWS::CloudFormation::Stack'
    Condition: CreateSagemakerNotebook
    Properties:
      TemplateURL: !Join
        - ''
        - - >-
            https://s3.amazonaws.com/aws-neptune-customer-samples/v2/cloudformation-templates/neptune-sagemaker-notebook-stack.json
      TimeoutInMinutes: 30
      Parameters:
        Env: !Ref Environment
        NotebookInstanceType: !Ref NotebookInstanceType
        NeptuneClusterEndpoint: !GetAtt
          - NeptuneStack
          - Outputs.DBClusterEndpoint
        NeptuneClusterPort: !GetAtt
          - NeptuneStack
          - Outputs.DBClusterPort
        NeptuneClusterVpc: !Ref VpcId
        NeptuneClusterSubnetId: !Select [0, !Ref SubnetIds]
        NeptuneClientSecurityGroup: !GetAtt
          - NeptuneStack
          - Outputs.NeptuneSecurityGroup
        NeptuneLoadFromS3RoleArn: !GetAtt
          - NeptuneStack
          - Outputs.NeptuneLoadFromS3IAMRoleArn
        StartupScript: !Ref NeptuneSagemakerNotebookStartupScript
Outputs:
  DBClusterId:
    Description: Neptune Cluster Identifier
    Value: !GetAtt
      - NeptuneStack
      - Outputs.DBClusterId
  DBSubnetGroupId:
    Description: Neptune DBSubnetGroup Identifier
    Value: !GetAtt
      - NeptuneStack
      - Outputs.DBSubnetGroupId
  DBClusterResourceId:
    Description: Neptune Cluster Resource Identifier
    Value: !GetAtt
      - NeptuneStack
      - Outputs.DBClusterResourceId
  DBClusterEndpoint:
    Description: Master Endpoint for Neptune Cluster
    Value: !GetAtt
      - NeptuneStack
      - Outputs.DBClusterEndpoint
  DBInstanceEndpoint:
    Description: Master Instance Endpoint
    Value: !GetAtt
      - NeptuneStack
      - Outputs.DBInstanceEndpoint
  SparqlEndpoint:
    Description: Sparql Endpoint for Neptune
    Value: !GetAtt
      - NeptuneStack
      - Outputs.SparqlEndpoint
  GremlinEndpoint:
    Description: Gremlin Endpoint for Neptune
    Value: !GetAtt
      - NeptuneStack
      - Outputs.GremlinEndpoint
  LoaderEndpoint:
    Description: Loader Endpoint for Neptune
    Value: !GetAtt
      - NeptuneStack
      - Outputs.LoaderEndpoint
  DBClusterReadEndpoint:
    Description: DB cluster Read Endpoint
    Value: !GetAtt
      - NeptuneStack
      - Outputs.DBClusterReadEndpoint
  DBClusterPort:
    Description: Port for the Neptune Cluster
    Value: !GetAtt
      - NeptuneStack
      - Outputs.DBClusterPort
  NeptuneLoadFromS3IAMRoleArn:
    Description: IAM Role for loading data in Neptune
    Value: !GetAtt
      - NeptuneStack
      - Outputs.NeptuneLoadFromS3IAMRoleArn
  NeptuneIamAuthUser:
    Description: IAM User for IAM Auth
    Value: !GetAtt
      - NeptuneStack
      - Outputs.NeptuneIamAuthUser
  EC2Client:
    Description: EC2 client Identifier
    Value: !GetAtt
      - NeptuneEC2Client
      - Outputs.EC2Client
  SSHAccess:
    Description: This is how you gain remote access to the EC2 client.
    Value: !GetAtt
      - NeptuneEC2Client
      - Outputs.SSHAccess
  VpcId:
    Description: VPC Id
    Value: !Ref VpcId
  NeptuneSagemakerNotebook:
    Condition: CreateSagemakerNotebook
    Description: Neptune Sagemaker Notebook Name
    Value: !GetAtt
      - NeptuneSagemakerNotebook
      - Outputs.NeptuneSagemakerNotebook
  NeptuneNotebookInstanceLifecycleConfigId:
    Condition: CreateSagemakerNotebook
    Description: Neptune Sagemaker Notebook Instance Lifecycle ConfigId
    Value: !GetAtt
      - NeptuneSagemakerNotebook
      - Outputs.NeptuneNotebookInstanceLifecycleConfigId
  NeptuneSecurityGroup:
    Value: !GetAtt NeptuneStack.Outputs.NeptuneSecurityGroup
    Export:
      Name: !Sub 'NeptuneSecurityGroup-NeptuneStack'
  PublicKey:
    Value: !GetAtt 'PrivateKey.PublicKey'
    Description: the public key, safe to expose
  PrivateKeyArn:
    Value: !GetAtt 'PrivateKey.Arn'
    Description: ARN of the private key in the Parameter Stor

neptune.yaml

https://gist.github.com/luzan/073fcbe9bc2980b59d09606a4dcdb75d

Posting it as link because of character limit on SO.

There are more, but there are the involved stacks that fail. Let me know if you need more for reference.

One thing is that you have Description as a property on the DBSubnetGroup, it should be DBSubnetGroupDescription though. — Robert Kossendey, Apr 08 '21 at 15:16
I actually tested with `DBSubnetGroupDescription`. Its same error. — Luzan Baral, Apr 08 '21 at 15:21
What happens when you eliminate all the "Name" tags and deploy. — Coin Graham, Apr 08 '21 at 19:57
@CoinGraham I just saw your answer, i was able to fixed it after removing all the Tags in multiple resources within neptune.yaml. — Luzan Baral, Apr 09 '21 at 06:09
You should be able to add back the tags other than "Name", perhaps it's reserved when using stacksets. — Coin Graham, Apr 09 '21 at 21:03
The reason is `cloudformation deploy` tags don't try to propagate stack tags to resources that support it and don't merge properly, so if you've already tagged the resources it will throw this error. — ThisGuyCantEven, Jul 20 '21 at 23:26

InvalidParameterCombination: "Duplicate tag key found in request" for Amazon Neptune Service CloudFormation

0 Answers0