Logstash : Is there a way to change some of the properties in document while migrating

Question

I have been migrating some of the indexes from self-hosted Elasticsearch to AmazonElasticSearch using Logstash. While migrating the documents, We need to change the field names in the index based on some logic.

Our Logstash Config file

input {
 elasticsearch {
 hosts => ["https://staing-example.com:443"]
 user => "userName"
 password => "password"
 index => "testingindex"
 size => 100
 scroll => "1m"
 }
}

filter {

}

output {
 amazon_es {
 hosts => ["https://example.us-east-1.es.amazonaws.com:443"]
 region => "us-east-1"
 aws_access_key_id => "access_key_id"
 aws_secret_access_key => "access_key_id"
 index => "testingindex"
}
stdout{
  codec => rubydebug
  }
}

Here it is one of the documents for the testingIndex from our self-hosted elastic search

{
    "uniqueIdentifier" => "e32d331b-ce5f-45c8-beca-b729707fca48",
         "createdDate" => 1527592562743,
     "interactionInfo" => [
         {
                        "value" => "Hello this is testing",
                        "title" => "msg",
            "interactionInfoId" => "8c091cb9-e51b-42f2-acad-79ad1fe685d8"
        },
         {
                        **"value"** => """"{"edited":false,"imgSrc":"asdfadf/soruce","cont":"Collaborated in  <b class=\"mention\" gid=\"4UIZjuFzMXiu2Ege6cF3R4q8dwaKb9pE\">@2222222</b>  ","chatMessageObjStr":"Btester has quoted your feed","userLogin":"test.comal@google.co","userId":"tester123"}"""",
                        "title" => "msgMeta",
            "interactionInfoId" => "f6c7203b-2bde-4cc9-a85e-08567f082af3"
        }
    ],
         "componentId" => "compId",

               "status" => [
                "delivered"
        ]
    },
           "accountId" => "test123",
       "applicationId" => "appId"
}

This is what we are expecting when documents get migrated to our AmazonElasticSearch

{
    "uniqueIdentifier" => "e32d331b-ce5f-45c8-beca-b729707fca48",
         "createdDate" => 1527592562743,
     "interactionInfo" => [
         {
                        "value" => "Hello this is testing",
                        "title" => "msg",
            "interactionInfoId" => "8c091cb9-e51b-42f2-acad-79ad1fe685d8"
        },
         {
                        **"value-keyword"** => """"{"edited":false,"imgSrc":"asdfadf/soruce","cont":"Collaborated in  <b class=\"mention\" gid=\"4UIZjuFzMXiu2Ege6cF3R4q8dwaKb9pE\">@2222222</b>  ","chatMessageObjStr":"Btester has quoted your feed","userLogin":"test.comal@google.co","userId":"tester123"}"""",
                        "title" => "msgMeta",
            "interactionInfoId" => "f6c7203b-2bde-4cc9-a85e-08567f082af3"
        }
    ],
         "componentId" => "compId",

               "status" => [
                "delivered"
        ]
    },
           "accountId" => "test123",
       "applicationId" => "appId"
}

What we need is to change the "value" field to "value-keyword" wherever we find some JSON format. Is there any other filter in Logstash to achieve this

You can look at these for the solution https://stackoverflow.com/questions/58538903/logstash-renaming-nested-fields-based-on-some-condition — Thilak, Nov 15 '19 at 08:11

score 0 · Answer 1 · answered Oct 21 '19 at 19:29

0

As documented in the Logstash website:

https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-rename

You can use the mutate filter, applying the rename function.

For example:

filter {
  mutate {
    replace => { "old-field" => "new-field" }
  }
}

For nested fields, you could just pass the path of the field:

filter {
  mutate {
    replace => { "[interactionInfo][value]" => "[interactionInfo][value-keyword]" }
  }
}

answered Oct 21 '19 at 19:29

Lodi

565
4
16

Hello, I am aware of mutate filter. what I need here is only change the field name if certain conditions met.is that is possible – Thilak Oct 22 '19 at 06:12
That depends where the data you need to apply conditional operators is. For example, if you have in your event a field `document_type`, you can use `if [document_type] == "process" mutate { ... }`. That, outside the `mutate` filter, but inside the `filter` block. How is that for you? – Lodi Oct 22 '19 at 11:35

Julien Poulin · Answer 2 · 2019-11-13T23:26:12.103

0

Try adding this to your filter:

filter {
  ruby {
    code => "event.get('interactionInfo').each { |item| if item['value'].match(/{.+}/) then item['value-keyword'] = item.delete('value') end }"
  }
}

edited Nov 13 '19 at 23:26

answered Oct 23 '19 at 20:32

Julien Poulin

12,737
10
51
76

.the value filed is a nested document. We are not sure how much value filed are present in each document. – Thilak Oct 31 '19 at 12:21
@Thilak, Sorry I didn't notice `interactionInfo` was an array. Please try my updated answer. – Julien Poulin Nov 13 '19 at 23:27
@Thilak Could you post an answer with your solution so that everybody can learn from it? – Julien Poulin Nov 14 '19 at 15:11
@Thilak Never mind, I found it: https://stackoverflow.com/q/58538903/65060 – Julien Poulin Nov 14 '19 at 15:13

Logstash : Is there a way to change some of the properties in document while migrating

2 Answers2