2

I am tried enabling the snapstart for my lambda function which just uses a simple java AWS SDK to connect to just a put a message to the queue. Without the snapstart the code works fine, however when its enabled, I get this error

Error while submitting the message with exception :Unable to load AWS credentials from any provider in the chain

I have attached the full access policy for the SQS and its working fine without the snapstart.

final Map<String, MessageAttributeValue> messageAttributes = new HashMap<>();
messageAttributes.put("testID", new MessageAttributeValue().withStringValue("id123").withDataType("String"));
final AmazonSQS sqs = AmazonSQSClientBuilder.standard().withRegion(Regions.EU_WEST_1).build();
final SendMessageRequest sendMessageStandardQueue = new SendMessageRequest().withQueueUrl("https://sqs.eu-west-1.amazonaws.com/111111/my-queue")
        .withMessageBody("Test message from lambda")
        .withMessageAttributes(messageAttributes);
sqs.sendMessage(sendMessageStandardQueue);

This code works fine without enabling snapstart on the lambda function. After enabling snapstart, I just added the following on the code.

 @Override
    public void beforeCheckpoint(org.crac.Context<? extends Resource> context) throws Exception {
        System.out.println("Before Checkpoint");
    }

    @Override
    public void afterRestore(org.crac.Context<? extends Resource> context) throws Exception {
        System.out.println("After Restore");
    }

Would you please help me to understand why exactly is it unable to load the credentials. ?

Jay
  • 31
  • 2
  • I've assumed that you're configuring an IAM role for this Lambda function because that's generally the best practice. Are you doing that? Also, did you try `AmazonSQSClientBuilder.defaultClient()` instead of `AmazonSQSClientBuilder.standard()`? – jarmod Dec 05 '22 at 22:43

3 Answers3

4

If you are relying on the AWS SDK's Environment Credentials Provider, then a rehydrated SnapStart Lambda Function will not find the AWS credentials. Per Using SnapStart with the AWS SDKs:

When SnapStart is activated, the Java runtime automatically uses the container credentials (AWS_CONTAINER_CREDENTIALS_FULL_URI and AWS_CONTAINER_AUTHORIZATION_TOKEN) instead of the access key environment variables. This prevents credentials from expiring before the function is restored.

So, I think this means:

  1. always provide IAM credentials to your Lambda function via an IAM role
  2. ensure that your code uses the default credentials provider chain

If that isn't the issue, then try to determine if your Lambda function relies on state that is not resilient to snapshot and restore operations. Consider using the AWS Lambda SnapStart Bug Scanner to investigate further.

In particular, the compatibility considerations include:

Temporary data

Some functions download or initialize ephemeral data, such as temporary credentials or cached timestamps, during the initialization phase. Refresh ephemeral data in the function handler before using it, even when not using SnapStart.

Also, a few restrictions at launch per the blog post:

  • it currently supports Java only and specifically the Corretto (java11) runtime
  • you cannot use Lambda SnapStart with larger ephemeral storage, Elastic File Systems, Provisioned Concurrency, or Graviton2
  • cached snapshots are removed after 14 days of inactivity
jarmod
  • 71,565
  • 16
  • 115
  • 122
  • Hi jarmod, read the compatability considerations, I am not intializing anything during the during the lambda start, the code above is just inside the handler so I would expect the lambda to connect to the SQS as before (without the snapstart, the same code works fine) – Jay Dec 03 '22 at 18:38
  • And you've double-checked that the Lambda has an assigned IAM role, of course. – jarmod Dec 03 '22 at 19:05
  • yes jarmod, I have assigned a role that has full access to SQS which made that code worked before enabling the snapstart – Jay Dec 03 '22 at 19:06
  • Yup, just wanted to double check it wasn't a second, parallel function. SnapStart requires Corretto 11 afaik. Unsure what else could be the cause. Are the runtime hooks you added making logs? Presume so. – jarmod Dec 03 '22 at 19:16
  • yes, I see the logs for that. The only thing is the I was not using cornetto 11 instead java 11. However I will change the java version to cornetto and check, thanks for the hint. – Jay Dec 03 '22 at 19:28
  • Hi @Jay , did you make any progress with this? – Martin Försterling Dec 05 '22 at 10:10
  • Hi Martin, I tried with the Amazon cornetto, but still I am having the same problem. Trying to understand what I am missing. – Jay Dec 05 '22 at 17:04
  • At this point, I think I would be tempted to try this with the most basic Java function that simply does STS GetCallerIdentity, prints out the identity, and nothing more. See if that works. Love the idea of Amazon Cornetto ;-) – jarmod Dec 05 '22 at 17:12
  • Hi Jarmod , it will not print out any identity as it could not get credentials from the provider chain to proceed further. – Jay Dec 05 '22 at 17:17
  • Sorry, I mean that I'd be tempted to go back to zero. Look at the AWS [sample](https://aws.amazon.com/blogs/compute/reducing-java-cold-starts-on-aws-lambda-functions-with-snapstart/) for SnapStart, for example, and build/deploy that. Does it work? If I then add SDK calls e.g. STS GetCallerIdentity, does that work? Just to take your current code/deployment out of the picture. I know that's some effort, however. Also be aware of [priming](https://aws.amazon.com/blogs/compute/reducing-java-cold-starts-on-aws-lambda-functions-with-snapstart/). – jarmod Dec 05 '22 at 18:06
  • In case it's of interest, I just noticed that there is a Serverless Land Office Hours [discussion on SnapStart](https://www.youtube.com/watch?v=Y5b8_KToeDY) scheduled for tomorrow at 1pm Eastern. – jarmod Dec 05 '22 at 20:53
  • @Jay I am facing the same issue. Did you figure out what was wrong? – EzyHoo Feb 12 '23 at 20:05
  • hi @EzyHoo, sorry for the late response, I have not figured out the solution for this yet. Please let me know if you have a solution for this issue. – Jay Mar 02 '23 at 22:03
3

How to place it working?

Using AWS SDK v2:

SqsClient sqs = SqsClient.builder()
    .credentialsProvider(SdkSystemSetting.AWS_CONTAINER_CREDENTIALS_FULL_URI
                                         .getStringValue()
                                         .isPresent()
        ? ContainerCredentialsProvider.builder().build()
        : EnvironmentVariableCredentialsProvider.create())
    .region(Region.of(SdkSystemSetting.AWS_REGION.getStringValueOrThrow()))
    .build();
mo7ty
  • 159
  • 10
2

For Snapstart environment, we will be using Container Credentials for SDK; for a normal Lambda environment, we can use Environment Credentials for SDK. One way to figure out which environment we are in is to check the container URI:

public AwsCredentialsProvider provideAwsCredentials()
return System.getenv("AWS_CONTAINER_CREDENTIALS_FULL_URI") == null ?
                EnvironmentVariableCredentialsProvider.create() : ContainerCredentialsProvider.builder().build();
EzyHoo
  • 301
  • 2
  • 14