Springboot, Micrometer and Aws Lambda function

Question

I'm trying to send metrics to AWS cloudwatch by using micrometer, however, I'm facing a problem with the AWS credentials.

ERROR i.m.c.CloudWatchMeterRegistry - error sending metric data. 
com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: 
[com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@b23c49d: Failed to connect to service endpoint: , com.amazonaws.auth.profile.ProfileCredentialsProvider@7edf67de: profile file cannot be null]
    at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:136)r 
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1257)r   
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:833)r 
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:783)r    
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)r 
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)r  
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)r   
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)r  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)r  
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)r  at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.doInvoke(AmazonCloudWatchClient.java:2587)r 
    at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:2554)r   
    at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:2543)r   
    at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.executePutMetricData(AmazonCloudWatchClient.java:2297)r 
    at com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient$27.call(AmazonCloudWatchAsyncClient.java:1215)r    
    at com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient$27.call(AmazonCloudWatchAsyncClient.java:1209)r    
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)r   
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)r at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)r    
    at java.base/java.lang.Thread.run(Unknown Source)r

The AmazonHttpClient is trying to retrieve the credentials using either the EC2ContainerCredentialsProviderWrapper or ProfileCredentialsProvider, but in the lambda environment the credentials are available through an execution role and also we have specific environment variables called AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

So, is there any ways to tell micrometer to use a different AwsCredentials provider, for example EnvironmentVariableCredentialsProvider?

can't you create a `@Bean` in a `@Configuration` file that provides that `CredentialProvider`? — Alberto Sinigaglia, Aug 21 '21 at 21:49
How are you creating the `CloudWatchMeterRegistry`? Custom `CloudWatchAsyncClient` should be passed into that, which can then use any credential provider you want. — Dejan Peretin, Aug 22 '21 at 10:00
@DejanPeretin I'm not creating on my own, it's provided as a bean in `CloudWatchExportAutoConfiguration`. Can you post an answer with the necessary to create a custom CloudWatchMeterRegistry? — Ele, Aug 22 '21 at 10:53
@DejanPeretin after creating my custom CloudWatchMeterRegistry bean, everything started to work. I tried that yesterday but didn't work because Terraform was not updating the code of my lambda function. Anyway, it's working now, thx. — Ele, Aug 22 '21 at 13:49

score 0 · Answer 1 · answered Aug 21 '21 at 21:50

0

Have you tried using a Configuration file to change the Bean for that?

@Configuration
public class ManualAWSCredentialProviderConfiguration {
  @Value("${AWS_ACCESS_KEY_ID}")
  protected String accessKey;

  @Value("${AWS_SECRET_ACCESS_KEY}")
  protected String secretKey;

  @Bean
  @Primary
  public AWSCredentialsProvider buildAWSCredentialsProviderManually() {
    return new AWSStaticCredentialsProvider(
      new BasicAWSCredentials(accessKey, secretKey)
    );
  }
}

answered Aug 21 '21 at 21:50

Alberto Sinigaglia

12,097
2
20
48

Unfortunately, it doesn't work, same problem. – Ele Aug 21 '21 at 22:17
@ele try to log the credentials that spring is assigning to those strings – Alberto Sinigaglia Aug 22 '21 at 00:40
The problem is that Micrometer is still using the EC2 and Profile credentials providers within the Cloudwatch client. – Ele Aug 22 '21 at 04:49

score 0 · Accepted Answer · answered Aug 22 '21 at 13:51

After some research, I was able to send the metrics by creating a custom CloudWatchMeterRegistry bean as follow:

@Bean
@Primary
public CloudWatchMeterRegistry customCloudWatchMeterRegistry(
        CloudWatchConfig config, Clock clock, AwsRegionProperties awsRegionProperties) {

    AmazonCloudWatchAsync amazonCloudWatchAsync = AmazonCloudWatchAsyncClient
            .asyncBuilder()
            .withCredentials(new EnvironmentVariableCredentialsProvider())
            .withRegion(awsRegionProperties.getStatic())
            .build();

    return new CloudWatchMeterRegistry(config, clock, amazonCloudWatchAsync);
}

As you can see, now I can configure a custom Credentials Provider, in my case the EnvironmentVariableCredentialsProvider.

Important hint: The name of the bean shouldn't be cloudWatchMeterRegistry because this class org.springframework.cloud.aws.autoconfigure.metrics.CloudWatchExportAutoConfiguration already has a declared bean with that name.

score 0 · Answer 3 · answered Aug 22 '21 at 14:07

When you add aws-java-sdk-sts to your dependencies (and thous having it on your classpath) will result in an extended provider chain. Then the execution role should get used.

For maven:

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-sts</artifactId>
    <version>1.12.52</version>
</dependency>

For gradle:

implementation group: 'com.amazonaws', name: 'aws-java-sdk-sts', version: '1.12.52'

Springboot, Micrometer and Aws Lambda function

3 Answers3