0

Dose anyone know if it is possible to collect ESX Performance Stats in LogStash/ELK stack? looking to see if I can collect ESXTOP data for use in the ELK stack.

TechGuyTJ
  • 792
  • 1
  • 13
  • 26

1 Answers1

1

Here's an example input, you would just need rsyslog or a logging tool to send logs to the input port (1514 in this example) and then filter the data:

Input

input {
        tcp {
                type => "VMware"
                port => "1514"
        }
}

Filter

filter {
    if "VMware" in [tags] {
            multiline {
                    pattern => "-->"
                    what => "previous"
            }
            grok {
                    break_on_match => true
                    match => [
                            "message", "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{SYSLOGPROG:syslog_program}: (?<message-body>(?<message_system_info>(?:\[%{DATA:message_thread_id} %{DATA:syslog_level} \'%{DATA:message_service}\'\ ?%{DATA:message_opID}])) \[%{DATA:message_service_info}]\ (?<syslog_message>(%{GREEDYDATA})))",
                            "message", "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{SYSLOGPROG:syslog_program}: (?<message-body>(?<message_system_info>(?:\[%{DATA:message_thread_id} %{DATA:syslog_level} \'%{DATA:message_service}\'\ ?%{DATA:message_opID}])) (?<syslog_message>(%{GREEDYDATA})))",
                            "message", "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{SYSLOGPROG:syslog_program}: %{GREEDYDATA:syslog_message}"
                    ]
            }
            syslog_pri { }
            date {
                    match => [ "syslog_timestamp", "YYYY-MM-ddHH:mm:ss,SSS" ]
                    timezone => "UTC"
            }
            mutate {
                    replace => [ "@source_host", "%{syslog_hostname}" ]
            }
            mutate {
                    replace => [ "@message", "%{syslog_message}" ]
            }
            if "Device naa" in [message] {
                    grok {
                            break_on_match => false
                            match => [
                                    "message", "Device naa.%{WORD:device_naa} performance has %{WORD:device_status}%{GREEDYDATA} of %{INT:datastore_latency_from}%{GREEDYDATA} to %{INT:datastore_latency_to}",
                                    "message", "Device naa.%{WORD:device_naa} performance has %{WORD:device_status}%{GREEDYDATA} from %{INT:datastore_latency_from}%{GREEDYDATA} to %{INT:datastore_latency_to}"
                            ]
                    }
            }
            if "connectivity issues" in [message] {
                    grok {
                            match => [
                                    "message", "Hostd: %{GREEDYDATA} : %{DATA:device_access} to volume %{DATA:device_id} %{DATA:datastore} (following|due to)"
                            ]
                    }
            }
            if "WARNING" in [message] {
                    grok {
                            match => [
                                    "message", "WARNING: %{GREEDYDATA:vmware_warning_msg}"
                            ]
                    }
            }
    }

}

lgroschen
  • 24
  • 4
  • if the ESX host is already syslogging to a syslog server and from there Logstash is picking it up, will this work in that context? – TechGuyTJ Mar 05 '15 at 20:54
  • Yes, you can use any type of logger just as long as you set the port that logstash will expect the data on and set udp/tcp if it's an option. Try this to verify log data on your receiving machine: `telnet ` – lgroschen Mar 05 '15 at 21:04
  • perfect, I will try this out and mark answered once I get it working. – TechGuyTJ Mar 05 '15 at 21:10
  • Hope it works, you may have to play with the filter if there are parts that aren't on the mark. grokparsefailure will be something you see if the filter failed – lgroschen Mar 05 '15 at 21:11
  • Did this end up working ok? – lgroschen Mar 10 '15 at 20:28
  • This didn't end up working for me. I did however take from it and get a more robust config. I will see if I can post what I have. – TechGuyTJ Mar 26 '15 at 17:37
  • Post it and I can test it with software that I work on that has a streamlined UI and has really good debugging and I could share that with you if you require. – lgroschen Mar 30 '15 at 17:38