2

We are building a system for realtime (target delay ~10 min) tracking of cloud infrastructure costs with graphs and notifications.
We've started with AWS but also plan to support Azure and GCP.
Currently our focus is to track costs of EC2 On-Demand instances. We are interested only in costs for the time that the instance was running, so we are not interested in tracking egress traffic, EBS usage or any other metrics related to ec2.
To track when each instance was started/stopped we use CloudWatch Events (EC2 Instance State-change Notification). Each state-change event contains an instanceId.
To get the prices we use the AWS Price List API.

To calculate the cost we need to find the prices of our instances in the list provided by the Price List API.
To be able to relate the instanceId to a price we need to know this info about the instance (we do only On-Demand instances for now):

  • 'Instance Type' - [m4.xlarge, m4.large, i3.xlarge, r5d.8xlarge, ...]
  • 'Location' - [us-east-1, us-east-2, us-west-1, ...]
  • 'Operating System' - [RHEL, Linux, Windows, SUSE]
  • 'License Model' - [No License required, Bring your own license]
  • 'Pre Installed S/W' - [-, SQL Std, SQL Ent, SQL Web]

Some pieces of this info (Instance Type and Location) along with imageId can be easily obtained by "describe-instances" command.
The rest are very problematic: Operating System, License Model, Pre Installed S/W.

The operating system is displayed in the AWS Console but there is no API to get it programmatically: image

There is a "describe-images" command that can provide some info about the image (by imageId) that the EC2 instance is using but there is no direct reference to the type of the OS:

{
    "Images": [
        {
            "Architecture": "x86_64",
            "CreationDate": "2019-06-19T21:59:15.000Z",
            "ImageId": "ami-0b898040803850657",
            "ImageLocation": "amazon/amzn2-ami-hvm-2.0.20190618-x86_64-gp2",
            "ImageType": "machine",
            "Public": true,
            "OwnerId": "137112412989",
            "State": "available",
            "BlockDeviceMappings": [
                {
                    "DeviceName": "/dev/xvda",
                    "Ebs": {
                        "Encrypted": false,
                        "DeleteOnTermination": true,
                        "SnapshotId": "snap-08091107f3acb12b2",
                        "VolumeSize": 8,
                        "VolumeType": "gp2"
                    }
                }
            ],
            "Description": "Amazon Linux 2 AMI 2.0.20190618 x86_64 HVM gp2",
            "EnaSupport": true,
            "Hypervisor": "xen",
            "ImageOwnerAlias": "amazon",
            "Name": "amzn2-ami-hvm-2.0.20190618-x86_64-gp2",
            "RootDeviceName": "/dev/xvda",
            "RootDeviceType": "ebs",
            "SriovNetSupport": "simple",
            "VirtualizationType": "hvm"
        }
    ]
}

There is also AWS Marketplace and it is able to show the prices for images in the UI but doesn't provide a public API and doesn't show the imageIds: enter image description here

Now our options to get the price of the instance are:

  • limit the range of supported images - use only those we now the OS for
  • get the price from the Cost and Usage Reports - but the reports are updated every 24 hours so there is no realtime
  • try to parse 'Description' provided by the "describe-images" command - but there is no guarantee that the description includes the correct OS name
  • try to interact with instances to determine their OS (AWS Systems Manager Inventory can be used for it) - this is complicated and requires software installed on instances
  • scrape data from the AWS Marketplace - obviously no one is going to do this
  • scrape data from the AWS console (that "Platform" field) - not really an option either
  • set tags on instances during their creation - obviously this is impossible if we already have a created instance

Please suggest other ways.

grabantot
  • 133
  • 6
  • 2
    Wouldn't it be easier to use the [AWS cost and usage reports](https://aws.amazon.com/aws-cost-management/aws-cost-and-usage-reporting/), which it can export hourly in CSV format to S3? This is hourly rather than every ten minutes, but are you really going to look at your costs every ten minutes? Compute cost tends to be outweighed by storage and bandwidth in many cases. – Tim Aug 15 '19 at 18:55
  • 1
    @Tim You can see I've listed Cost and Usage Reports as one of the options. The thing with them is that even if you choose hourly granularity the reports are updated once a day. In our system we have a UI that hides all of AWS complexity and uses only a limited set of AWS resources. We want it to be simple for our clients and for them to be able to track their costs transparently in realtime. 1 hour delay may be works for us unfortunately for Cost and Usage Reports it is 24 hours. – grabantot Aug 16 '19 at 05:27

0 Answers0