1

Problem Statement: I want to fetch cost metrics, filtered by linked account and grouped by service, for my AWS account using boto3. Then, I want to post this data to Elasticsearch (ES) and visualize that data on Kibana in a pie chart which shows break down of cost grouped by service (EC2, S3 etc.). I also want to be able to filter the data based on time and visualize it using a time series bar chart. These visualizations need to be on one dashboard. Note that the timestamp field has been deprecated by ELK which is adding to complexity. I'm currently using the get_cost_and_usage() service (boto3) to retrieve the data and dumping (PUT call) it to ES using the ES API.

get_cost_and_usage() gives me a response back which is grouped by service which is not easily readable in ES. How do I get it in a format which can be easily visualized in Kibana? Also, I need to be able to filter by time upto 6 months back. How do I do that with the @timestamp field being deprecated in ES now. Are there any sample Kibana visualizations/dashboards for this which I can leverage without reinventing the wheel? I know that the AWS Cost Billing dashboard could be used for the same, but I need to get this working in Kibana.

My request payload is as shown below:

POST URL: https://ce.us-east-1.amazonaws.com

{
"TimePeriod": {
"Start":"2018-08-01",
"End": "2018-09-01"
},
"Granularity": "MONTHLY",
"Filter": {      
"Dimensions": {
  "Key": "LINKED_ACCOUNT",
  "Values": [
    "<aws_account_id>"
  ]
}
},
"GroupBy":[
{
  "Type":"DIMENSION",
  "Key":"SERVICE"
},
{
  "Type":"TAG",
  "Key":"Environment"
}
],
"Metrics":["BlendedCost", "UnblendedCost", "UsageQuantity"]
}
buddhapiano
  • 63
  • 1
  • 4
  • Have you heard of https://aws.amazon.com/aws-cost-management/aws-cost-explorer/ ? It does a lot of this out of the box and it's at no additional cost to your AWS account. I know that doesn't answer your question - just wanted to let you know. – rwisch45 Sep 29 '18 at 21:03
  • @rwisch45 yes I've heard of costexplorer. Infact I'm using the costexplorer API to get the cost data. I figured out a way to format the data in key value pairs and post it to ES – buddhapiano Oct 12 '18 at 18:08
  • If you are using a not AWS Software to manage your cluster try to integrate Kubecost https://kubecost.com/ – Miguel Conde Jun 28 '19 at 17:34

0 Answers0