Mix nested query with scripted filter

Question

Context

We have data about malware in elasticsearch. We must analyse it with kibana. For each sample (malware), we have some SSL certificates. These are nested fields. I'm new to elasticsearch and have difficulties to find what I want in the official documentation.

What I want

I want to be able to create a chart (ie: Line chart) respecting these rules:

Simple count on the Y Axis
Certificate file name on the X Axis
Show only Subject == Issuer

The simplified mapping

{
    "mappings":{  
        "sample":{
            "properties":{
                "static":{
                        "x509Certificates":{
                            "type":"nested",
                            "properties":{
                                "Status":{
                                    "type":"long"
                                },
                                "FileName":{
                                    "type":"keyword"
                                },
                                "Serial":{
                                    "type":"long"
                                },
                                "Signature":{
                                    "type":"keyword"
                                },
                                "PublicKey":{
                                    "type":"keyword"
                                },
                                "NotValidBefore":{
                                    "type":"date",
                                    "format":"epoch_millis"
                                },
                                "NotValidAfter":{
                                    "type":"date",
                                    "format":"epoch_millis"
                                },
                                "Subject":{
                                    "type":"keyword"
                                },
                                "Issuer":{
                                    "type":"keyword"
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

Things I already tried

Not using nested objects

It's not working because there is no "join". The chart is created, but the information worth nothing.

Using the kibana nested fork

https://github.com/homeaway/kibana/tree/nestedSupport-5.2 It's actually helping. Atleast we can generate a graph with the nested fields. But it did not completely resolve the problem.

Using scripted query in the search bar on the top of the chart

The query was similar to:

 {
   "bool": {
     "must": {
        "script": {
           "script": {
              "inline": "doc['Subject'].value == doc['Issuer'].value",
              "lang": "painless"
              }
           }
        }
     }
 }

I'm not sure at all, but it's like if it search for at least one of both value being equals between all the certificates.

Using scripted fields

I have created a simple boolean scripted field similar to:

doc['Subject'].value == doc['Issuer'].value

And created a query for the boolean being true. It was partially working.

Nested query

It really looks like what I want BUT, I don't know how to say Issuer==Subject with this method.

My question

What is the easiest way to solve my problem knowing that this is not our only use case that need nested scripted filters?

Don't hesitate to ask for clarification!

score 2 · Accepted Answer · answered Mar 02 '17 at 10:18

2

What if you have your scripted field as such, since both Subject and Issuer are type of keyword:

def newfield = "";

if((doc['Subject'].value).equals(doc['Issuer'].value)){
   newfield= "matched";
}else{
   newfield= "not matched";
}

And then maybe you could use the above scripted field by applying a filter within your graph, as:

scriptedFieldName:"matched"

Hope this helps!

answered Mar 02 '17 at 10:18

Kulasangar

9,046
5
51
82

What is the difference between == and .equals()? – rm4 Mar 02 '17 at 16:03
`==` always just compares two references for non-primitives. The `equals()` method compares the "value" inside String instances irrespective if the two object references refer to the same String instance or not. Hope I made myself clear. – Kulasangar Mar 02 '17 at 16:48
It explain why it was not working before..... I prefer to use a boolean value for this type of use case. But the point is to really use .equals() instead ==. Thank you! – rm4 Mar 03 '17 at 13:36