9

Summary: I want to attach a TTL field with the logs in logstash and send them over to the Elastic search.

I have already gone through the documentation but could not get much of it, since it is not very clear.

This is my config file in logstash.

input {
  stdin {
    type => "stdin-type"
  }
}

output {
  stdout { debug => true debug_format => "json"}
  elasticsearch {}
}

Now suppose that for each log that is read, I want to attach a TTL with it for say, 5 days.

I know how to activate the TTL option in elastic search. But What changes will I have to make in the elastic search configuration files is not very clear to me. The documentation asks to look for the mappings folder, but there is none in the elastic search download folder.

Looking for an expert help.

javanna
  • 59,145
  • 14
  • 144
  • 125
user2359303
  • 261
  • 2
  • 7
  • 15

1 Answers1

12

Have a look here if you want to put the mapping on file system. You have to go to the config folder and create here a folder called mappings, and another one with the name of the index within mappings. Since logstash creates by default an index per day, you'd better use the _default name for the folder, so that the mapping will be applied to all indexes. The file that you create under that folder must have the name of the type you want to apply the mapping to. I don't remember exactly what type logstash uses, thus I would use the _default_ mapping definition. Just call the file _default_.json and put the following content in it:

{
    "_default_" : {
        "_ttl" : { "enabled" : true }
    }
}

As you can see the name of the type must appear in both the filename and in its content.

Otherwise, you could avoid putting stuff on file system. You could create an index template containing your custom mapping, like the following:

{
    "template" : "logstash-*",
    "mappings" : {
        "_default_" : {
            "_ttl" : { "enabled" : true }
        }
    }
}

The mapping will then be applied to all the indices whose name matches the template pattern. If you use the _default_ mapping definition the mapping will be applied as default to all the types that are going to be created.

javanna
  • 59,145
  • 14
  • 144
  • 125
  • @javanna...thanx for your help... But I want to know where do I put all these files mentioned by you. Say I go by the first method, I have to create 2 folders, mappings and _default under config, right? Then where do I specify the TTL thing? – user2359303 May 11 '13 at 12:35
  • Will I have to create a json file? If yes, then what will it contain and what will be its name? The doc says [mapping-name]. Could you please clarify all this? I am really confused – user2359303 May 11 '13 at 12:37
  • Thanks. Just that the folder _default has to be under mappings and not directly under the config folder, right? – user2359303 May 12 '13 at 04:22
  • @ javanna It still doesnt work... pls help me with this http://stackoverflow.com/questions/16516353/ttl-elastic-search-not-working – user2359303 May 13 '13 at 14:21
  • I have created the _default_.json(which is template file) file and I have copied the below code in _default_.json { "template" : "logstash-*", "mappings" : { "_default_" : { "_ttl" : { "enabled" : true } } } } Now where to put this file in elastic search and where to give the entry for that template so that it will call automatically ?? – ashishl Dec 02 '16 at 10:04