How to generate an AWS profile in `~/.aws/config` for use in a CodeBuild Project

Question

I'm working with a tool called dbt and the database authentication method for the tool utilizes IAM. Unfortunately, IAM profiles do not exist when CodeBuild projects are built because it utilizes Instance Profiles instead. Beacause of this I am unable to connect to my database.

Referencing this question, I tried running aws sts get-caller-identity in the project to see if I was able to get some of the values I need returned, but it returned

botocore.exceptions.ProfileNotFound: The config profile (***) could not be found

Does anyone have idea on how to generate my own ~/.aws/config within a CodeBuild project?

edit: the tool uses boto3 to generate temporary credentials here: https://github.com/fishtown-analytics/dbt/blob/9d00c000720d17c42a4fa08a26b75bd500cc857f/plugins/redshift/dbt/adapters/redshift/connections.py#L101-L123

but it does not seem to be able to generate those credentials within a CodeBuild project.

edit:

buildspec.yml

version: 0.2

env:
  variables:
    MODELS_REPO: dbt-dev
    PYTHON_VERSION: 3.8
  parameter-store:
    AWS_ENVIRONMENT: "/cloudformation/environment"
    AWS_PROFILE: "/cloudformation/environment"
    CODEARTIFACT_COMPANY: "/codeartifact/company"
    GITHUB_OWNER: "/github/owner"
    GITHUB_PERSONAL_ACCESS_TOKEN: "/secret/github/token"
    GITHUB_USER: "/github/user"

phases:
  install:
    runtime-versions:
        python: "${PYTHON_VERSION}"
    commands:
      - pip install -r projects/${PROJECT_NAME}/requirements.txt
      - ./projects/${PROJECT_NAME}/.aws/phases/install.sh
  pre_build:
    commands:
      - ./projects/${PROJECT_NAME}/.aws/phases/pre_build.sh
  build:
    commands:
      - ./projects/${PROJECT_NAME}/.aws/phases/build.sh
  post_build:
    commands:
      - ./projects/${PROJECT_NAME}/.aws/phases/post_build.sh

cache:
  paths:
    - /root/.cache/pip
    - /root/.cache/pip/**/*
    - ~/.cache/pip
    - ~/.cache/pip/**/*

Could you clarify what do you mean that instance profile doesn't exist? You add permissions to your CB using CB role. — Marcin, Sep 22 '20 at 23:44
I understand that, but I believe I need some sort of hacky solution to generate a config at `~/.aws/config` so that I can use the `dbt` tool. This tool requires the ability to generate database credentials with IAM. — metersk, Sep 22 '20 at 23:55
You can run [aws configure set](https://docs.aws.amazon.com/cli/latest/reference/configure/set.html) with the values of parameters you want to set. — Marcin, Sep 23 '20 at 00:02

score 3 · Accepted Answer · answered Oct 06 '20 at 12:15

Following script should work for your use-case:

apt install jq -y
creds=$(aws sts get-session-token)

AWS_ACCESS_KEY_ID=$(echo $creds | jq '.Credentials.AccessKeyId')
AWS_SECRET_ACCESS_KEY=$(echo $creds | jq '.Credentials.SecretAccessKey')
AWS_SESSION_TOKEN=$(echo $creds | jq '.Credentials.SessionToken')

aws configure --profile $AWS_PROFILE set region "us-east-1"
aws configure --profile $AWS_PROFILE set output "json"
aws configure --profile $AWS_PROFILE set aws_secret_access_key "$AWS_SECRET_ACCESS_KEY"
aws configure --profile $AWS_PROFILE set aws_access_key_id "$AWS_ACCESS_KEY_ID"
aws configure --profile $AWS_PROFILE set aws_session_token "$AWS_SESSION_TOKEN"

You can change the region for your needs.

How to generate an AWS profile in `~/.aws/config` for use in a CodeBuild Project

1 Answers1