How do I tell Elasticsearch to get the column names from the first line of a file?

Question

I just started using Elasticsearch/logstash.

I have 3 different files with a common id. Each file contains the column names on the first line, for example:

header1,header2,header3,header4
1234,data2,data3,data4
1235,data2,data3,data4
1236,data2,data3,data4

How can I tell Elasticsearch to get the first line as column names?

Also, how can I do some research in Elastic using the common id between the files, for example q:column=data group by id?

score 2 · Answer 1 · answered Jul 12 '18 at 23:50

The other answers are outdated as of CSV filter plugin version 3.0.8 (maybe earlier). Now you can do this:

filter { csv { autodetect_column_names => true } }

In case it's not working as expected, you'll have to add pipeline.workers: 1 into your logstash.yml file.

See: https://github.com/logstash-plugins/logstash-filter-csv/issues/65

score 1 · Answer 2 · answered Feb 08 '16 at 16:57

1

Looks like you have csv data. Logstash provides a csv filter, but it doesn't handle header rows.

There is a new csv codec, but it's listed as not ready for production.

answered Feb 08 '16 at 16:57

Alain Collins

16,268
2
32
55

score 0 · Answer 3 · edited Jan 25 '17 at 07:13

0

you can add the below mentioned code in the .confgig file . You have to explicitly mention the column names in the config file .

filter {
csv {
    columns => [
      "YearMonth",
      "ProjectCode",
      "EmpNo",
      "RevenueCreditUnit",
      "Revenue",
      "dtLoad"

    ]
    separator => ","
    remove_field => ["message"]
    }
}

Keep searching , there might be a better way .

edited Jan 25 '17 at 07:13

Danny_ds

11,201
1
24
46

answered Sep 02 '16 at 08:17

Arun Naudiyal

361
4
4

How do I tell Elasticsearch to get the column names from the first line of a file?

3 Answers3