How do I replace a field in logstash

Question

I'm doing an insert from Logstash into ElasticSearch. My problem is that I used a template in ES to lay out the data types, and I am sometimes getting values from Logstash that are null values (or dashes) when I've declared in ES that they should be doubles.

So sometimes, ES is getting a '-' instead of something like "2342", and it is rejecting it and causing an error. Now, if I can replace the '-' with the word 'null', ES works fine.

How do I do this? I assume it works with the ruby filter. I need to be able to replace the '-' fields with null when appropriate.

EDIT:

I was asked for sample configs.

So, for example, say the below config is logstash, which will then send data to ES:

filter {
    if [type] == "transaction" {
        match => ["message", "%{BASE16FLOAT:ts}\t%{IP:orig_ip}\t%{NOTSPACE:orig_port}" ]
    }
}

Now my ES template is saying:

"transaction" : {
    "properties" :
    {
        "ts" : {
            "format" : "dateOptionalTime",
            "type" : "date"
        },
        "orig_ip" : {
            "type" : "ip"
        },
        "orig_port" : {
            "type" : "long"
        },
   }
}

So if I throw a data set like either of these, it passes:

{"ts" : "123456789.123234", "orig_ip" : "10.0.0.1", "orig_port" : "2342" }
{"ts" : "123456789.123234", "orig_ip" : "10.0.0.1", "orig_port" : null }

I get a success. But, the following [obviously] fails:

{"ts" : "123456789.123234", "orig_ip" : "10.0.0.1", "orig_port" : "-" }

How can I ensure that the "-" (with quotes) gets changed to a null?

Can you show your logstash config, your mapping type and a sample document? — Val, Aug 07 '15 at 03:50
Possible duplicate of [How do I replace a string in a field in Logstash](https://stackoverflow.com/questions/42092394/how-do-i-replace-a-string-in-a-field-in-logstash) — RASG, Jun 13 '17 at 11:58

score 0 · Answer 1 · answered Aug 07 '15 at 13:40

0

If you amend your template by specifying "ignore_malformed": true in your orig_port long field, it should work.

"transaction" : {
    "properties" :
    {
        "ts" : {
            "format" : "dateOptionalTime",
            "type" : "date"
        },
        "orig_ip" : {
            "type" : "ip"
        },
        "orig_port" : {
            "type" : "long"
            "ignore_malformed": true     <---- add this line
        }
   }
}

answered Aug 07 '15 at 13:40

Val

207,596
13
358
360

So this will work, but I'd still like an alternative with a filter, that will change the values for me. I'll keep looking, and if I find something, I'll post it. – jasonmclose Aug 09 '15 at 14:57

How do I replace a field in logstash

1 Answers1