AWS Kinesis Firehose is not sending data to Elasticsearch....IAM permissions?

Question

So I have everything ready and working good except for the last step which sending data from Firehose to Elasticsearch.

This is the error I get in Kinesis Firehose Elasticsearch service logs:

Error received from Elasticsearch cluster. {"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [indices:data/write/bulk] and User [name=arn:aws:iam::917877325894:role/firehose_delivery_role, backend_roles=[arn:aws:iam::917877325894:role/firehose_delivery_role], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [indices:data/write/bulk] and User [name=arn:aws:iam::917877325894:role/firehose_delivery_role, backend_roles=[arn:aws:iam::917877325894:role/firehose_delivery_role], requestedTenant=null]"},"status":403}

And this is the IAM policy I have attached to it(which was made by Firehose itself)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeVpcs",
                "ec2:DescribeVpcAttribute",
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeNetworkInterfaces",
                "ec2:CreateNetworkInterface",
                "ec2:CreateNetworkInterfacePermission",
                "ec2:DeleteNetworkInterface"
            ],
            "Resource": "*"
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::test-kinesis-backup-mydna",
                "arn:aws:s3:::test-kinesis-backup-mydna/*"
            ]
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "lambda:InvokeFunction",
                "lambda:GetFunctionConfiguration"
            ],
            "Resource": "arn:aws:lambda:us-west-2:917877325894:function:%FIREHOSE_DEFAULT_FUNCTION%:%FIREHOSE_DEFAULT_VERSION%"
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "es:DescribeElasticsearchDomain",
                "es:DescribeElasticsearchDomains",
                "es:DescribeElasticsearchDomainConfig",
                "es:ESHttpPost",
                "es:ESHttpPut"
            ],
            "Resource": [
                "arn:aws:es:us-west-2:917877325894:domain/test-dynamodb",
                "arn:aws:es:us-west-2:917877325894:domain/test-dynamodb/*"
            ]
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "es:ESHttpGet"
            ],
            "Resource": [
                "arn:aws:es:us-west-2:917877325894:domain/test-dynamodb/_all/_settings",
                "arn:aws:es:us-west-2:917877325894:domain/test-dynamodb/_cluster/stats",
                "arn:aws:es:us-west-2:917877325894:domain/test-dynamodb/test_dynamodb*/_mapping/",
                "arn:aws:es:us-west-2:917877325894:domain/test-dynamodb/_nodes",
                "arn:aws:es:us-west-2:917877325894:domain/test-dynamodb/_nodes/stats",
                "arn:aws:es:us-west-2:917877325894:domain/test-dynamodb/_nodes/*/stats",
                "arn:aws:es:us-west-2:917877325894:domain/test-dynamodb/_stats",
                "arn:aws:es:us-west-2:917877325894:domain/test-dynamodb/test_dynamodb*/_stats"
            ]
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:us-west-2:917877325894:log-group:/aws/kinesisfirehose/test_dynamodb:log-stream:*"
            ]
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "kinesis:DescribeStream",
                "kinesis:GetShardIterator",
                "kinesis:GetRecords",
                "kinesis:ListShards"
            ],
            "Resource": "arn:aws:kinesis:us-west-2:917877325894:stream/%FIREHOSE_STREAM_NAME%"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt"
            ],
            "Resource": [
                "arn:aws:kms:us-west-2:917877325894:key/%SSE_KEY_ID%"
            ],
            "Condition": {
                "StringEquals": {
                    "kms:ViaService": "kinesis.%REGION_NAME%.amazonaws.com"
                },
                "StringLike": {
                    "kms:EncryptionContext:aws:kinesis:arn": "arn:aws:kinesis:%REGION_NAME%:917877325894:stream/%FIREHOSE_STREAM_NAME%"
                }
            }
        }
    ]
}

I tried to add all different policies but could not get it to work and I keep getting same exact error message.

Any suggestions?

Are you sure that that your ES domain is called `test-dynamodb`? Also what is the policy on the ES domain? — Marcin, Jun 12 '20 at 02:34
@Marcin yes it is called test-dynamodb. and the policy is { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": [ "*" ] }, "Action": [ "es:*" ], "Resource": "arn:aws:es:us-west-2:917877325894:domain/test-dynamodb2/*" } ] } — Data_sniffer, Jun 12 '20 at 17:02
Is this policy auto generated? Usually for `es:` you would have `es:*`. Similar for Principle. — Marcin, Jun 12 '20 at 22:56

score 5 · Accepted Answer · answered Dec 30 '20 at 15:25

I just ran into the same issue. My problem was that I accidentally assigned the firehose ARN to ES role mapping, instead of the IAM ARN.

Elastic Search > Security > firehose_delivery_role > mapped users > backend role > {{ this needs to be the IAM firehose ARN, not the firehose ARN itselt }}

Incorrect: arn:aws:firehose:us-east-1:000000012345:deliverystream/workshop-firehose

Correct: arn:aws:iam::000000012345:role/service-role/KinesisFirehoseServiceRole-workshop-fire-us-east-1-1609335111111

Before finding the error, I also tried adding [indices:data/write/bulk] and [indices:data/write/bulk*] to both the cluster and index permissions of the ES role... but that did not help.

Hope this helps anyone else struggling with similar issue.

score 0 · Answer 2 · answered Aug 11 '21 at 10:21

I also faced the same issue. I set up AWS elastic search with Cognito authentication. Adding firehose iam role arn to ES access policy solved the issue

{
"Version": "2012-10-17",
"Statement": [
{
  "Sid": "ESAccess",
  "Effect": "Allow",
  "Principal": {
    "AWS": [
      "arn:aws:iam::<accountID>:role/firehose-assume-role",
      "arn:aws:iam::<accountID>:role/cognito-auth-role"
    ]
  },
  "Action": "es:*",
  "Resource": [
    "arn:aws:es:<region>:<accountID>:domain/es-domain/*",
    "arn:aws:es:<region>:<accountID>:domain/es-domain"
  ]
  }
 ]
}

AWS Kinesis Firehose is not sending data to Elasticsearch....IAM permissions?

2 Answers2