How to configure aws credentials to setup cloudwatch with fluentbit

Question

I need to send logs to cloudwatch using fluentbit, from the application hosted on my local system, but i am unable to configure the aws credentials for fluent bit to send logs to cloudwatch. It will be of great help if anyone can help me with the same. Some of the logs are as follows:-

[aws_credentials] Initialized Env Provider in standard chain
[aws_credentials] Failed to initialized profile provider: $HOME not set and AWS_SHARED_CREDENTIALS_FILE not set.
[aws_credentials] Not initializing EKS provider because AWS_ROLE_ARN was not set
[aws_credentials] Initialized EC2 Provider in standard chain
[aws_credentials] Not initializing ECS Provider because AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is not set
[aws_credentials] Sync called on the EC2 provider
[aws_credentials] Init called on the env provider
[aws_credentials] Init called on the EC2 IMDS provider
[aws_credentials] requesting credentials from EC2 IMDS

score 0 · Answer 1 · answered Nov 09 '20 at 15:24

0

Any standard way to pass credentials should work here:

export environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
or create ~/.aws/credentials, may use aws configure prompt for this.

answered Nov 09 '20 at 15:24

Max Lobur

5,662
22
35

Hey max, i have added these but it is still giving me the above mentioned error. – Raghu kapur Nov 09 '20 at 16:24
ok then let's try to address what it wants: `export $HOME=your_home_pah` and `export AWS_SHARED_CREDENTIALS_FILE=$HOME/.aws/credentials` – Max Lobur Nov 09 '20 at 17:23
1

hey hii...sorry for the late reply, even doing what you have mentioned is giving the same error. – Raghu kapur Nov 11 '20 at 04:34

score 0 · Answer 2 · answered Jul 22 '21 at 15:53

During my testing the variable weren't enough same for the credentials file. I installed AWS cli configure it with the keys and now it works as expected. I am working with containers and the AWS cli add extra size that I don't need so if anyone knows a way to do it without it. that would be awesome.

score 0 · Answer 3 · answered Mar 09 '23 at 18:28

I ran into this same problem on my server when running the fluent-bit service. Installing the AWS CLI and setting the config and credentials in ~/.aws/config and ~/.aws/credentials still wasn't enough.

My solution was to follow this Stackoverflow post, and create a service override.conf file for the fluent-bit service where the ENV variables for HOME, AWS_CONFIG_FILE, and AWS_SHARED_CREDENTIALS_FILE are all explicitly defined.

The override.conf file looked like this:

[Service]
Environment="HOME=/root"
Environment="AWS_CONFIG_FILE=/root/.aws/config"
Environment="AWS_SHARED_CREDENTIALS_FILE=/root/.aws/credentials"

After setting that and restarting the fluent-bit service, fluent-bit successfully sent logs to AWS CloudWatch, and I stopped getting the Failed to initialized profile provider error.

How to configure aws credentials to setup cloudwatch with fluentbit

3 Answers3