DiskSpaceUtilization metrics for EC2 instance in python boto script showing empty response

Question

I want to get the datapoints for DiskSpaceUtilization metrics but the response for get_metrics_statitics is empty.

The get_metrics_statitics function works perfectly for other metrics i.e. CPUUtilization, MemoryUtilization. But the same code doesn't work for DiskSpaceUtilization.

I tried the below code:

import sys
from datetime import datetime as dt, timedelta
import boto3

metricdictionary = {}
metricdictionary['DiskSpaceUtilization'] = 'System/Linux,Percent'

ec2_resource = boto3.resource("ec2")
cloudwatch = boto3.client("cloudwatch")

date = dt.today() - timedelta(days=1)
year = date.year
month = date.month
day = date.day

response = cloudwatch.get_metric_statistics(Namespace='System/Linux', 
                            MetricName='DiskSpaceUtilization', 
                            Dimensions=[{'Name': 'InstanceId', 
                            'Value': 'i-0a22a230c4dae4195', }],                                
                            StartTime=dt(year, month, day, 00, 00, 00),
                            EndTime=dt(year, month, day, 23, 59, 59),
                            Period=3600, 
                            Statistics=['Average', 'Minimum', 'Maximum'],
                            Unit='Percent')

print response

Thanks in advance.

Fabio Manzano · Answer 1 · 2018-09-04T14:12:09.033

2

That's because you need to specify two extra dimensions, in addition to InstanceId: Filesystem and MountPath.

There's a similar issue reported in AWS Developer Forums. Check the note in case you don't want to hardcode these two dimensions:

One way to avoid using instance-specific filesystem dimensions is to modify the mon-put-instance-data.pl script and just comment out the part that adds Filesystem and MountPath dimensions, leaving only InstanceId dimension for the reported disk metrics. When making these modifications, try running mon-put-instance-data.pl script from the command line with --verify --verbose options to see what data will be actually sent over to Amazon CloudWatch service.

EDIT: In case you want to get these two dimensions using Python, you can do something similar to this:

#!/usr/bin/env python

import subprocess

mounts = {}

for line in subprocess.check_output(['mount', '-l']).split('\n'):
    parts = line.split(' ')
    if len(parts) > 2:
        mounts[parts[2]] = parts[0]

print mounts

Reference: https://askubuntu.com/questions/189987/get-device-with-mount-point

edited Sep 04 '18 at 14:12

answered Sep 03 '18 at 18:06

Fabio Manzano

2,847
1
11
23

I wan't diskspace utilization datapoints using boto3 in python but I don't know how many file system does instance have So how do I get the filesystem for instance So that I can modify the get_metrics_statitics function to read DiskSpace datapoints.Any idea – Vikas Satpute Sep 04 '18 at 09:50
Two ways: (1) Comment out filesystem and mountpath from mon-put-instance-data.pl, as I said in the post. (2) Get all mounting points from your current instance. This post tells a few ways to accomplish (e.g. findmnt or /proc/mounts): https://unix.stackexchange.com/questions/24182/how-to-get-the-complete-and-exact-list-of-mounted-filesystems-in-linux – Fabio Manzano Sep 04 '18 at 11:47
The post that you shared is showing getting mounted file system in linux but my question is how to get mounted file system using boto in python. – Vikas Satpute Sep 04 '18 at 13:16
It doesn't seems that boto API provides this info... I looked at the [official API documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#volume) an also a few StackOverflow threads ([1](https://stackoverflow.com/questions/5251057/using-boto-to-find-to-which-device-and-ebs-volume-is-mounted) and [2](https://stackoverflow.com/questions/41318892/get-volume-information-associated-with-instance)). But it is a simple command that you can execute in Python. I'll update the thread with the code. – Fabio Manzano Sep 04 '18 at 13:44
I am running this on one EC2 server and reading clouwatch data for all other EC2 server in production So the command is " mount -l " should be run on every single instances to get file system mounted on it by instance-id. Is there is any way to run the command on remote server by their instance-id. – Vikas Satpute Sep 05 '18 at 07:33
Have you tried to set a cron schedule in user-data? That way it will always run your scripts at EC2 launch. Also, have you tried to edit the pl script (mon-put-instance-data.pl)? It may be easier: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html#using_put_script – Fabio Manzano Sep 05 '18 at 12:40
Yeah you are correct if I add it to cronjob to each instance but we have more than 100 instances up and running so it becomes more manual work. And the other thing is if I edited and comment the part that adds Filesystem and MountPath dimensions then how do I understood respective filesystem metrics to collect the datapoints. – Vikas Satpute Sep 05 '18 at 14:40
I was wondering if there's a default mountpath and filesystem values. Isn't that the same for all your VMs? Like the same AMI? I guess a short term solution would be do hard code these dimensions. – Fabio Manzano Sep 05 '18 at 14:55
Thanks Fabino I had remodify the function now it works perfectly. The filesystem's and mountpoint are same for every VM. I am self answering the question have a look at get_metric_statitics function. Thanks for helping – Vikas Satpute Sep 06 '18 at 12:00
Perfect, I'm happy to hear that! Thanks. – Fabio Manzano Sep 06 '18 at 12:31

score 0 · Accepted Answer · edited May 04 '22 at 15:38

I modified the get_metric_statistics function and it works perfectly. For Disk Space utilization we need to add extra two dimensions, i.e. FileSystem and Mountpath.

response = cloudwatch.get_metric_statistics(
    Namespace=namesp, MetricName='DiskSpaceUtilization',
    Dimensions=[
        {'Name': 'InstanceId', 'Value': instance['InstanceId'], },
        {'Name': 'Filesystem', 'Value': name_of_file_system},
        {'Name': 'MountPath', 'Value': MountPath_for_filesystem}
    ],
    StartTime=dt(year, month, day, 00, 00, 00),
    EndTime=dt(year, month, day, 23, 59, 59),
    Period=3600, Statistics=['Average', 'Minimum', 'Maximum'],
    Unit='Percent')

DiskSpaceUtilization metrics for EC2 instance in python boto script showing empty response

2 Answers2