Internal Server Error when uploading threat lists to AWS Guard Duty

Question

I'm currently using CloudFormation Templates to automate uploading threat intelligence sets to AWS Guard Duty. After weeks of successful testing, I'm now getting internal server errors whenever I try to upload a threat intelligence set. This happens both trying to upload via the console or programatically via Boto3. When testing on my personal account, I'm still able to upload threat lists just fine. Any suggestions for how to proceed on this? I'm a bit of a loss since it's obviously hard to debug server errors from the client side

Answer 1

I ended up just talking to AWS support, and I'll paste the answer to help anyone else out with a similar issue. tldr; it's a known bug that probably won't occur in production environments and has a relatively simple fix

Before we talk about the fix, let me explain how GuardDuty, IAM and S3 interact (You may already know this). When a user call "CreateThreatIntelSet" API then on the backend a "PutRolePolicy" [IAM] is called which allows service role "AWSServiceRoleForAmazonGuardDuty" to perform "GetObject" [S3] to the S3 object hosting the threat list. More details:

https://docs.aws.amazon.com/guardduty/latest/ug/guardduty_upload_lists.html

So keeping above information in mind, you would should notice that the IAM API call of "PutRolePolicy" which is called by GuardDuty is failing due to the policy size of 10240 bytes restriction which in-turn failed "CreateThreatIntelSet" with internal error.

Please see: IAM limits: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-limits.html

So I believe "CreateThreatIntelSet" was called multiple times which created multiple inline policies in AWSServiceRoleForAmazonGuardDuty service role.

Having duplicate inline policies is not an issue but due to duplicate policies we are hitting an IAM policy limits. You can navigate to below location to confirm this on your end:

https://console.aws.amazon.com/iam/home?region=us-east-1#/roles/AWSServiceRoleForAmazonGuardDuty

A simple fix for the issue would be to remove duplicate inline policy in AWSServiceRoleForAmazonGuardDuty service role and then create the threat list again.

...

Action Plan: 1- Please remove duplicate inline policy in AWSServiceRoleForAmazonGuardDuty service role and then create the threat list again. 2- If you update your threat list regularly then we would recommend to use UpdateThreatIntelSet API instead of deleting the exciting threat list then creating a new one: https://docs.aws.amazon.com/guardduty/latest/ug/update-threat-intel-set.html

This answer is actually slightly incorrect: I couldn't find a way to remove inline policies, as boto3 complained that users weren't allowed to edit service-linked roles. So I just ended up deleting the role and recreating it (minus the 50 inline policies), which worked