1

I have setup an Amazon linux AMI with the Kinesis Agent installed and configured to send the logs over to Firehose. The EC2 instance has been attached an IAM role with KinesisFirehoseFullAccess permission. However I am receiving the inadequate permissions error while the data is being sent over.

I know that I have provided the highest level of IAM Kinesis permissions but I am facing a blank wall now. I will, of course, trim the permissions down later but I first need to get this proof of concept working.

From the AWS Firehose, did a test send to the S3 bucket. This worked OK. Created logs via the Fake Log Generator. I then ran the service. Service is up and running.

User: arn:aws:sts::1245678012:assumed-role/FirstTech-EC2-KinesisFireHose/i-0bdf3adc7a4d97afa is not authorized to perform: firehose:PutRecordBatch on resource: arn:aws:firehose:ap-southeast-1:1245678012:deliverystream/firsttech-ingestion-weblogs (Service: AmazonKinesisFirehose; Status Code: 400; Error Code: AccessDeniedException;

localhost (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent. Agent: Progress: 900 records parsed (220430 bytes), and 0 records sent successfully to destinations. Uptime: 840058ms

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
Ochen Ao
  • 93
  • 2
  • 11
  • I would start by logging into the instance and testing the permissions on the IAM Role assigned to the instance. For example, use the AWS CLI to run `aws firehose list-delivery-streams ...` to confirm that it has Firehose permissions. The try a manual `aws firehose put-record-batch ...` command to see whether the permissions are correct. Also, in reading [Writing to Amazon Kinesis Data Streams Using Kinesis Agent - Amazon Kinesis Data Streams](https://docs.aws.amazon.com/streams/latest/dev/writing-with-agents.html), I'd suggest giving `kinesis:PutRecord` permissions too, to see if it helps. – John Rotenstein Apr 01 '20 at 03:50
  • No joy. I created a new Role and added the PutRecord and PutRecordBatch permissions, still the same...:( – Ochen Ao Apr 02 '20 at 07:04
  • Did you try sending the data via the AWS CLI as suggested in my comment above? What were the results? – John Rotenstein Apr 02 '20 at 07:21
  • Have an update.... – Ochen Ao Apr 02 '20 at 11:22
  • after much tinkering i have managed to get the data to be parsed, transformed and sent to Kinesis Firehose albeit not the way i wanted the setup to be. Here's what i did. I had already created an EC2 instance profile role and attached the KinesisFirehoseFullAccess permission to the role and attached it to the instance but it was still giving out that error. So:- 1. Added KinesisFullAccess permission to the EC2 role, still no joy. 2. Removed the Kinesis and Firehose Full Access permissions and attached a custom policy with Kinesis PutRecord and PutRecordBatch permissions, no joy. – Ochen Ao Apr 02 '20 at 11:23
  • 3. Added the AWS access KEY ID and Access Secret in the agent.conf file, no joy. 4. Removed the attached role from the EC2 instance, works now ! What i don't understand is why it wouldn't work with IAM roles cause really i don't want to be using credentials to configure, or at least avoid it wherever possible. Speaking of which i have run into another issue, i don't see the data coming over to the S3 bucket... – Ochen Ao Apr 02 '20 at 11:23
  • however, if try to send the test demo data from the kinesis console, works ... – Ochen Ao Apr 02 '20 at 11:24
  • Sorry, spoke too soon. I can see the data now on the S3 bucket but when i open it, it shows this. So still a perms issue somewhere? AccessDenied Access Denied – Ochen Ao Apr 02 '20 at 11:51
  • the size of the files in the S3 bucket is 1 MB as i expected. But when i open it, all is see id the Access Denied error output with the request id. II am just wondering since i used my AWS Access Key ID and secret, do i need to feed that to kinesis somewhere somehow? Maybe in the IAM permissions? – Ochen Ao Apr 02 '20 at 14:17
  • I have fixed the nagging issue myself. On the ec2 instance, i configured the AWS ID and SECRET using the aws command line. It's a shame though that i couldn't get it to work using IAM. I'll keep looking for a solution for this but if you have any solution, happy to try that out. Thanks..... Let me know on this thread. – Ochen Ao Apr 02 '20 at 18:06

2 Answers2

0

I got this working for the aws kinesis agent to send data to a kinesis data stream just in case anyone else here has issues. I had the same issues after attaching the correct iam role and policy permissions to an ec2 instance that needed to send records to a kinesis data stream. I just removed the references to aws firehose in the config file. You do not need to use keys embedded in the ec2 instance itself, the iam role is sufficient.

enter image description here

user5664690
  • 1
  • 1
    linking to code(config ect ..) in an image is not recommended, as a result this post will probably get down voted, to avoid this please just include the code in the question – Nigel Savage Aug 27 '21 at 20:35
  • While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. – Stanley Aug 28 '21 at 05:25
0

Make sure you create the Kinesis Firehose with from console to ensure that all required IAM accesses are instantiated by default.

Next in your instance ensure your agent.json is correct from:

{
  "cloudwatch.emitMetrics": true,
  "flows": [
    {
      "filePattern": "/var/log/pathtolog",
      "deliveryStream": "kinesisstreamname"    }
  ]
}

Make sure the EC2 instance has the necessary permissions to send the data to Kinesis:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "cloudwatch:PutMetricData",
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "firehose:PutRecord",
                "firehose:PutRecordBatch"
            ],
            "Resource": "arn:aws:firehose:us-east-1:accountid:deliverystream/deliverstreamname"
        }
    ]
}

Also make sure you kinesis agent can collect data from any directory in the instance - the easiest way to do this is by adding the agent to the sudoers group.

sudo usermod -aG sudo aws-kinesis-agent-user
Leon Africa
  • 509
  • 6
  • 11