AWS DotNet SDK Error: Unable to get IAM security credentials from EC2 Instance Metadata Service

Question

I use an example from here in order to retreive a secret from AWS SecretsManager in c# code.

I have set credentials locally via AWS CLI, and I am able to retreive secret list using AWS CLI command "aws secretsmanager list-secrets".

But c# console app fails with an error:

> Unhandled exception. System.AggregateException: One or more errors occurred. (Unable to get IAM security credentials from EC2 Instance Metadata Service.)
 ---> Amazon.Runtime.AmazonServiceException: Unable to get IAM security credentials from EC2 Instance Metadata Service.
   at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.FetchCredentials()
   at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentials()
   at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentialsAsync()
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at AWSConsoleApp2.GetSecretValueFirst.GetSecret() in D:\Work\Projects\Training\AWSConsoleApp2\AWSConsoleApp2\GetSecretValueFirst.cs:line 53
   at AWSConsoleApp2.Program.Main(String[] args) in D:\Work\Projects\Training\AWSConsoleApp2\AWSConsoleApp2\Program.cs:line 11

When I change original constructor call

IAmazonSecretsManager client = new AmazonSecretsManagerClient();

with adding inherited parameter of type AWSCredentials

IAmazonSecretsManager client = new AmazonSecretsManagerClient(new StoredProfileAWSCredentials());

it works fine.

Class StoredProfileAWSCredentials is obsolete but it works to use it. I use libraries that work without errors on the other machines and I cannot change them.

I use credentials for user that belongs to Administrators group and has full access to SecretsMnager. Region has set properly in c# code, profile is default.

Any ideas? Thanks for advance

This a fallback error related to a default configuration not being available, and if unsuccessful from the Instance Profile service on an EC2 instance. It is documented on the `AmazonCognitoIdentityProviderClient` — davidcarr, Jan 08 '23 at 14:55

score 17 · Answer 1 · answered Jun 28 '21 at 14:06

17

I've run into this issue a number of times, but have not been able to resolve it using the above solutions.

What has worked for me is explicitly setting my AWS profile using the AWS_PROFILE environment variable and setting it to the profile I want to use.

Today I ran into this issue again, where even that didn't work. What eventually solved it was setting the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

I dread the day where I run out of alternative ways to supply credentials to AWS.

answered Jun 28 '21 at 14:06

Nino van der Mark

622
1
9
19

1

Thank you!! Providing AWS_PROFILE environment variable in Visual Studio 2019 project properties (in the Debug tab) immediately helped authenticate using my custom profile. My sample app is .NET Core 3.1. The AWS CLI profile is stored in %USERPROFILE%\.aws\credentials file on Windows 10). Previously, I unsuccessfully attempted to set the profile by using the code `Amazon.AWSConfigs.AWSProfileName = "Client200";` which resulted in the aforementioned exception. – timmi4sa Sep 23 '21 at 17:12

score 15 · Accepted Answer · edited Mar 16 '23 at 12:30

15

I had the same issue, and here is how I fixed it in my development environment

I created an AWS profile using the AWS Explorer extension for Visual Studio. This is also called the AWS Toolkit for Visual Studio.
Once the profile is set up the credentials are passed in using the profile.

Please note that the profile accessing the AWS Secrets Manager secret must have the proper authorization to do so. This AWS documentation will help get you in the right direction.

edited Mar 16 '23 at 12:30

SovietFrontier

2,047
1
15
33

answered May 28 '20 at 09:24

Vikas Sharma

471
5
8

Thank you. It helped. I had profile file created by AWS CLI with needed data though, SDK could not read it apparently – Robinson May 28 '20 at 23:09
12

Point to note here, the user profile accessing the key manager should have a valid security group assigned for the Secrets manager. What? – Vlado Pandžić Jul 29 '20 at 09:05
1

A link to the extension would be nice. – Trevor Sep 09 '22 at 15:27

Marius · Answer 3 · 2021-11-12T07:37:33.333

Since AWS SDK credentials configuration is causing a lot of headache, I'll throw in some context. First of all, if you are using dotnet core, use the AWSSDK.Extensions.NETCore.Setup package (https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/net-dg-config-netcore.html), which will respect your appsettings.json.

{
  "AWS": {
    "Region": "eu-west-1",
    "Profile": "theprofileyouwantouse"
  }
}

csproj:

  <ItemGroup>
    <PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="3.7.1" />
    <PackageReference Include="AWSSDK.SecurityToken" Version="3.7.1.71" />
  </ItemGroup>

Example:

var config = host.Services.GetService<IConfiguration>();
var options = config.GetAWSOptions();
using var client = options.CreateServiceClient<IAmazonSecurityTokenService>();
var result = await client.GetCallerIdentityAsync(new Amazon.SecurityToken.Model.GetCallerIdentityRequest { });

This will try to pick up encrypted credentials in ~/AppData/Local/AWSToolkit and secondly based on your shared config file (~/.aws/config). As of november 2021, it does not utilize aws_access_key_id, aws_secret_access_key, aws_session_token in the version 1 shared credentials file (~/.aws/credentials)*

Next, if the roles you are assuming are AWS SSO, you need the following packages in your csproj file:

    <PackageReference Include="AWSSDK.SSO" Version="3.7.0.94" />
    <PackageReference Include="AWSSDK.SSOOIDC" Version="3.7.0.94" />

*If you happen to have invertedly added your credentials to your shared credentials file (~/.aws/credentials) as [profile myprofile] instead of just [myprofile] the SDK will not behave as you expected, so delete that. If your credentials file is fine, then you don't have to touch it, but keep in mind that the SDK will noe use the cached credentials if any found in that file.

Now, the author does not use the AWSSDK.Extensions.NETCore.Setup package, which means that we are getting a slightly different credentials resolving path. Most importantly: appsettings.json is not respected, this means you must specify the profile you want to use differently, for example by using the AWS_PROFILE environment variable.

Secondly, we are landing directly in the FallbackCredentialsFactory.cs which does this when resolving credentials:

            CredentialsGenerators = new List<CredentialsGenerator>
            {
#if BCL
                () => new AppConfigAWSCredentials(),            // Test explicit keys/profile name first.
#endif
                () => AssumeRoleWithWebIdentityCredentials.FromEnvironmentVariables(),
                // Attempt to load the default profile.  It could be Basic, Session, AssumeRole, or SAML.
                () => GetAWSCredentials(credentialProfileChain),
                () => new EnvironmentVariablesAWSCredentials(), // Look for credentials set in environment vars.
                () => ECSEC2CredentialsWrapper(proxy),      // either get ECS credentials or instance profile credentials
            };

Now the last step in resolving credentials "ECSEC2" has a fallback which returns this:

DefaultInstanceProfileAWSCredentials.Instance

Which leads us to the error which the author sees.

Summary:

If you are not using AWSSDK.Extensions.NETCore.Setup, specify the profile using an ENV-variable in launch.json or launchSettings.json if you are going to use the default constructor like the author
Rember to add the AWS SSO packages if needed

You state that the credentials in `~/.aws/credentials` aren't used, and this seems to be accurate, but then how do we specify credentials via a profile? The `aws config --profile xxx` command sets the access key and secret key in the `~/.aws/credentials` file, and so they aren't picked up. — Mark, Jan 10 '23 at 17:39
It depends on the flow, if you are talking about AWS SSO, the only thing needed is the access-token which is stored in ~/.aws/sso/cache, which the cli (and most SDKs) use. The operation performed by the CLI and most SDKs is the same as the one described here: https://aws.amazon.com/premiumsupport/knowledge-center/sso-temporary-credentials/ — Marius, Jan 11 '23 at 08:00

score 8 · Answer 4 · answered Jul 14 '22 at 19:31

8

If anyone is using docker-compose and getting this error, I added this to my docker-compose.override.yml file and it was able to read my credentials

volumes:
  - ~/.aws/:/root/.aws:ro

answered Jul 14 '22 at 19:31

smcg

135
1
8

On Windows add this to docker-compose.override.yml file volumes: - ${USERPROFILE}/.aws:/root/.aws:ro # AWS credentials – Brett Aug 27 '23 at 11:37

score 7 · Answer 5 · answered Jun 19 '20 at 14:57

7

Same issue and resolved by deleting $HOME/.aws/config and credentials files and recreating with AWS CLI.

In my case I was switching laptops from Windows to a new MBP. I had setup my new environment by copying the .aws directory files and confirmed that AWS CLI worked correctly. Confusingly the dotnet SDK failed with same errors.

answered Jun 19 '20 at 14:57

Kyle

71
1
1

Same as you, I was getting this error in a Visual Studio project after copying the `.aws` folder to a new laptop. I deleted the folder, used `aws configure` to recreate the folder and files with the default section, I copied my other sections (i.e. profiles other than default) manually and it worked. – AJ Dhaliwal Mar 18 '22 at 11:12

score 7 · Answer 6 · edited Sep 13 '21 at 07:31

7

I had the same issue, and resolved it by changing the name of the AWS profile in Visual Studio to default.

edited Sep 13 '21 at 07:31

Dharman

30,962
25
85
135

answered Sep 13 '21 at 07:26

Rukshan Dangalla

2,500
2
24
29

score 6 · Answer 7 · answered Mar 18 '21 at 22:01

6

Run the following command and follow the prompt using the data provided by AWS:

aws configure

answered Mar 18 '21 at 22:01

Julian Espinel

2,586
5
26
20

score 1 · Answer 8 · answered Dec 04 '20 at 12:48

The question is not exactly my problem, but it's the first hit on google so I figured I'd chip in just in case.

I got the exact above error when issuing

dotnet lambda list-layers

It seems like the dotnet cli uses the AWS_PROFILE variable and does not default to AWS_DEFAULT_PROFILE. In my company, the AWS_DEFAULT_PROFILE is mapped to an identity provider, thus I do not manage different access with different profiles and the default profile is empty. As a workaround, run your command like this

AWS_PROFILE=$AWS_DEFAULT_PROFILE dotnet lambda list-layers

This way the CLI will use the correct credentials.

score 1 · Answer 9 · answered Jun 29 '21 at 19:07

1

make sure you have the latest version of EC2config installed https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/UsingConfig_Install.html

Thanks

answered Jun 29 '21 at 19:07

Joe

2,381
1
17
17

I installed EC2Launch and Ran my service and it returned the Keys from SecretsManager. But only once. Later it is again throwing same exception – Raj Chaurasia Sep 04 '21 at 10:49

score 1 · Answer 10 · answered Jul 29 '21 at 20:17

Just add env variables in control panel AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. The actual value of either is not important. I have set them both to a space (' '). Don't know why it works but it works. It does seem to take longer to log in. It seems that instead of going to the buggy flow, the SDK tries to use the env vars, fails and about after 30 seconds or so logs in as required.

Tested it on two different Win10 PCs with no AWS CLI installed or any AWS profile configured. The issue was recreated 100% and the described w/a fixed it.

score 1 · Answer 11 · answered Sep 13 '21 at 23:22

I was deploying to a dot net core web application to an on prem server over IIS and had the same exact issue. No matter what I did the application would not recognize my credentials configured via AWS CLI (aws configure).

I ended up setting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with my keys via the Windows environment variables and restarting the server.

The following article was very helpful in understanding the AWS SDK credential loading Client Factory https://www.stevejgordon.co.uk/credential-loading-and-the-aws-sdk-for-dotnet-deep-dive

score 1 · Answer 12 · answered Oct 12 '21 at 16:02

1

I had the same issue and it turned out to be because I had AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN uppercased in my credentials file. Changing the keys to lowercase solved it for me.

answered Oct 12 '21 at 16:02

Nikolas Rodrigues Teixeira

11
1

score 1 · Answer 13 · answered Dec 27 '21 at 19:53

In my case the config and credentials files were set up correctly in C:/Users//.aws folder so they should have been found by default. However, on a previous project I had set up different credentials (no longer valid) in C:/Users//AppData/Local/AWSToolkit referred to in AWS documentation as the AWS SDK Store. The SDK store is always checked first and then falls back to the default user credentials file. See the following: https://aws.amazon.com/blogs/developer/referencing-credentials-using-profiles/. The simplest solution in my case was simply to delete the files in the AWSToolkit folder. As an alternative I could have set up the SDK Store correctly.

score 1 · Answer 14 · answered Feb 24 '22 at 14:33

1

In the project defaults.json, verify the profile value. in my case it was empty "profile": "". After setting the profile name, was able to publish

answered Feb 24 '22 at 14:33

Balaji

21
2

score 0 · Answer 15 · answered May 17 '22 at 20:57

I had the same problem in .NET core 5 with AWS and I solved it by :

what I had :

I had config and credentials files in C:\Users\.aws.

In StartUp.cs after initialized AWS options I added:

#if Debuge
   options.Profile="default";
   options.ProfileLocations="C:\\Users\\.aws\\credentials";
#endif

score 0 · Answer 16 · edited Jul 29 '22 at 16:18

I had the same issue:

Amazon.Runtime.AmazonServiceException: 'Unable to get IAM security credentials from EC2 Instance Metadata Service.'

I was working with Dot Net Core Microservice, I got this error.

Solution - I removed the AWS credentials path which was mentioned in all the different setting files like appsettings.Debug.json and appsettings.Development.json.

This AWS credentials path should only be mentioned in the appsettings.json file. Remove it from all other files.

score 0 · Answer 17 · answered Jan 15 '23 at 17:41

Posting this here as it is at least the 2nd time I've caused the error via a self-inflicted misconfiguration.

VS 2022
AWS Toolkit 1.38.0.0

None of the existing answers worked, but they did point to a few configuration issues. One or more of these configuration settings did/could cause the error:

AWS Toolkit Explorer

When using the AWS Toolkit Explorer to define the profile, make sure the correct profile is selected.

Personally, I no longer define a default profile. I use named profiles for all credentials. This is especially useful when dealing with multiple profiles (I have at least 10+).

Environmental Variables

I've found the most consistent way to run an AWS process locally (or as a service) is to set the AWS_PROFILE Environmental Variable. Example:

Environment.SetEnvironmentVariable( "AWS_PROFILE", "<profle_name>", EnvironmentVariableTarget.Process );

In this particular case, I used an old profile name that had been changed. In other words, if you're using an Environmental Variable to set the AWS_PROFILE, make sure the profile name is correct.

When using the Environmental Variable approach, do not set an Environmental Variable for AWS_REGION. The AWS_PROFILE defines the region when the profile is correctly defined.

Other Thoughts...

When configured correctly, there should be no need for any other AWS environmental variable. Not even empty variables for:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN

score 0 · Answer 18 · answered Feb 19 '23 at 12:48

0

Need to add the AWS access key and secret key in the visual studio extension, same as in the image.

answered Feb 19 '23 at 12:48

Rodrigo Furlaneti

11
3

score 0 · Answer 19 · answered May 02 '23 at 21:27

If none of the other answers work for you and you're attempting to use SSO, make sure you're using at a minimum the .NET Framework 4.5 assemblies. The 3.5 assemblies have some logic compiled out of them that support SSO.

The error this question is citing is what happens whenever all of the main credential locations in the search order fail to find credentials for whatever reason; EC2 instance metadata is the last place that is searched (and will fail if you're not on an EC2 instance).

score 0 · Answer 20 · answered Jun 02 '23 at 06:16

I think this Exception is from new AmazonDynamoDBClient() constructor where it couldn't find security credentials.

One way to fix this problem is by leveraging AWSSDK.Extensions.NETCore.Setup and AWSSDK.SecurityToken nuget packages.

When you register DynamoDB, instead of doing

.AddSingleton<IAmazonDynamoDB, AmazonDynamoDBClient>()

with the packages, you can now do

.AddDefaultAWSOptions(webApplicationBuilder.Configuration.GetAWSOptions())
.AddAWSService<IAmazonDynamoDB>()

If you have an authenticated profile in your .aws/credentials file and you have added the profile in appsettings.json like

"AWS": {
  "Profile": "yourProfile"
}

It should fix this error.

score 0 · Answer 21 · answered Aug 10 '23 at 13:28

Set the AWS default profile on the machine using the command prompt.

aws configure --profile "default"

AWS Access Key ID and AWS Secret Access Key - You have already noted these details after the user is created

Default region name - You have to use the same region name where your DynamoDB table is created.

Default output format - Leave it blank

(This is working in my case)

AWS DotNet SDK Error: Unable to get IAM security credentials from EC2 Instance Metadata Service

21 Answers21

AWS Toolkit Explorer

Environmental Variables

Other Thoughts...

Linked