1

I would like to know if it is possible to evaluate and optimize AWS spending (specifically, EC2 spending) without having access to account billing info?

Long story short, we do not have the ability to view account billing dashboards / metrics due to a company policy. (Such a permission would allow us to see other billing metrics that are sensitive).

However, I can get a monthly snapshot of our AWS bill, with a breakdown for each service.

I know that our EC2 spending is very high. But without having the ability to drill into this data to get more insights, how do I optimize? I have some theories on where most of our spending is coming from (EMR/spark compute nodes), but I can't know for sure unless I can measure it.

Is there a way for me to get compute hours usage for EC2, without going through billing dashboards / APIs? I imagine if I can get the total hours for a machine, and multiply it by the hourly rate, I should be able to generate a cost estimate myself?

RE: close vote:

I think this is not off topic. I am asking for a way to programmatically measure the compute hours for AWS virtual machines (EC2). This seems to be in scope for stackoverflow.

James Wierzba
  • 16,176
  • 14
  • 79
  • 120
  • Without getting detailed billing report, it will be more or less a wild guess as you will have no idea which EC2s or any other resources are costing how much without identifying them based on the tags. If you have single resource of a service like one RDS instance or one DynamoDB table, you will be OK but for multiple resources of a service will be a guess. – Asdfg Jul 24 '19 at 14:49
  • While there is an organization account that consolidates all the accounts under it regarding billing, each account has it's own billing, so either (1) ask the root permission on your account (and not access to the consolidated account) or (2) ask the IAM permissions to access billing on your account under your user. If you have any issues with it as it is a "shared" account, ask to be moved to a different account and get access to the "billing" service on your account. Once there, you can see the spend on everything related to your account alone, and you won't have access to other metrics. – Dvir669 Jul 24 '19 at 20:54

2 Answers2

2

One suggestion, that I do not know if fits your company policy, is:

  1. Enable to generate a billing Detailes Billing CSV in an S3 Bucket with restricted access (https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/billing-getting-started.html)
  2. Create a Lambda Function that runs when the CSV is updated and process the CSV to filter only the data you should have access (https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html)
  3. Save a new CSV with only the allowed information in a new S3 Bucket with more access allowed.

Hope this approach makes sense in your company policies. If it does, I can give more details on how to setup.

filipebarretto
  • 1,842
  • 1
  • 13
  • 27
0

As a starting point, you could use AWS CloudWatch and check the metrics usage for each of your EC2 instances (such as CPU usage, memory, and so on...). This gives you a first overview of how your EC2 are using their resources. You can start adjusting from there.

Also there are basic steps to already save money with EC2 (i.e. using Reserved instances).

Then if your company don't give you direct access to aws billing through the console (I know the pain :/ I've experienced the same), then I really would suggest to build a website or whatever lambda function, give it's role access to aws cost explorer and use the API to retrieve billing data.

Here is a good benefit for that: using the API you can order by different values compared to the AWS console where you can just order by one single property. This way you can generate for instance a report per service but per environment too. I did that for my company splitting per stack but also per environment (DEV/TEST/PROD). This way I can generate a precise billing per month or "on the fly" for the current month up to current date (I can attach a screenshot later -> I have no access to it atm).

A review of your code will ensure the security officer or whoever is in charge that you only access your billing data.

TaiT's
  • 3,138
  • 3
  • 15
  • 26