EnhanceIO Telegraf `filestat` plugin - Super GLOB pattern not working - Make telegraf recursively look for all files

Question

Telegraf v1.0.1 (git: master 26acdc9231efde105510fe5df3da7519bc4f42f7)

Telegraf service is running successfully sudo service telegraf status shows telegraf is running [OK].

I'm using Wavefront's SaaS based monitoring solution for showing Telegraf data or setting up various other things (alerts, dashboards).. it works.

Overview: When you install Telegraf, it creates its main config file at /etc/telegraf/telegraf.conf and a user can put other configurations under /etc/telegraf/telegraf.d/*.conf (files).

I have /etc/telegraf/telegraf.d/extra-inputs-plugins.conf and in this file, I have the following contents (as you see, it's using filestat inputs plugin) and the following configuration works:

## Telegraf filestat plugin
[[inputs.filestat]]
  files = ["/var/run/*/*.pid","/var/run/*.pid"]

On some database servers, I have installed EnhanceIO (for more info look here: https://github.com/stec-inc/EnhanceIO

Once EnhanceIO is installed, you'll get a folder structure like this:

ubuntu@MyTestCluster-1a-db2-i-0cf6u98b136b211ba:~$ find /proc/enhanceio
/proc/enhanceio
/proc/enhanceio/data_cache
/proc/enhanceio/data_cache/config
/proc/enhanceio/data_cache/io_hist
/proc/enhanceio/data_cache/errors
/proc/enhanceio/data_cache/stats
/proc/enhanceio/version

To configure Telegraf's filestat plugin to catch/look for /proc/enhanceio/data_cache/config file, I can add that or /proc/enhanceio/data_cache/* in my configuration (but doing this way, the solution won't be scalable i.e. what if I want telegraf to pick all files under /proc folder.

The plugin doc / comments section says:

  ## These accept standard unix glob matching rules, but with the addition of
  ## ** as a "super asterisk".

So, I tried the following configuration to look for every file (recursively):

[[inputs.filestat]]
  files = ["/var/run/*/*.pid","/var/run/*.pid","/proc/*"]

The above resulted in the following output when I run: $ telegraf --config-directory=/etc/telegraf -test|grep filestat|grep -v '/var/run/'|grep enhance (actually /proc/enhanceio is a folder).

> filestat,host=MyTestCluster-1a-db2-i-0cf6u98b136b211ba,file=/proc/enhanceio exists=1i,size_bytes=0i 1485548956000000000

Then, I tried using the ** approach but I got NOTHING?

[[inputs.filestat]]
  files = ["/var/run/*/*.pid","/var/run/*.pid","/proc/**"]

$ telegraf --config-directory=/etc/telegraf -test|grep filestat|grep -v '/var/run/'|grep enhance
2017/01/27 20:31:38 I! Using config file: /etc/telegraf/telegraf.conf
$

I tried, almost all glob patterns (like: /proc/enhanceio/*/*, /proc/enhanceio/*/**, /proc/enhanceio/**/* or /proc/enhanceio/**/**) but it just didn't catch any files under /proc/enhanceio tree.

Why filestat plugin's SUPER GLOB pattern didn't work at all when I tried the above patterns?

How can I make the filestat plugin catch all files under /proc tree?

PS: I know giving /proc/enhanceio/data_cache/* will work if I want to catch config file under that directory (at just that level).

Answer 1

As per Cameron Sparr's comment and testing on this, she couldn't reproduce the above situation with the following example but the examples I mentioned in my post were valid as well in not catching the super glob patterns.

Per her comment, it seems like: /proc is a very special "filesystem" that is actually a "file" mapping to particular kernel parameters and metrics. Glob and path matching may not work in this area as you might expect.

% ls -R /tmp/test
/tmp/test:
enhance/  foo.log

/tmp/test/enhance:
bar.log  nested/

/tmp/test/enhance/nested:
foo.file
then with this config:

[[inputs.filestat]]
  files = ["/tmp/test/**.log", "/tmp/test/**.file"]
  ## If true, read the entire file and calculate an md5 checksum.
  md5 = false
I was able to find all files

% telegraf --config ~/gd/ws/telegraf.conf --input-filter filestat --output-filter discard --test
* Plugin: inputs.filestat, Collection 1
> filestat,file=/tmp/test/enhance/bar.log,host=tyrion size_bytes=4i,exists=1i 1485988684000000000
> filestat,file=/tmp/test/foo.log,host=tyrion size_bytes=0i,exists=1i 1485988684000000000
> filestat,file=/tmp/test/enhance/nested/foo.file,host=tyrion exists=1i,size_bytes=0i 1485988684000000000