1

I'm trying to fetch data from MySQL and push it to ElasticSearch using Logstash and I'm having trouble creating a config file for Logstash that suits my need

I'm trying to achieve this result:

{
   "products":[
      {
         "id":1,
         "deals":[
            {
               "id":5,
               "options":[
                  {
                     "id":3
                  },
                  {
                     "id":8
                  }
               ]
            }
         ]
      }
   ]
}

In MySQL, each of these has its own table, meaning that

Product -> Deal (ONE => MANY) | Deal -> Deal Option(ONE => MANY)

To combine them all, I have a MySQL View that would LEFT JOIN those tables so I can process everything using LogStash

Here is my current LogStash Configuration

filter {
  aggregate {
    task_id => "%{id}"
    code => "
    map['id'] ||= event.get('id')

    map['deals'] ||= []    
    map['deals'] << {'id' => event.get('deal_id')}


    event.cancel()
    "
    push_previous_map_as_event => true
    timeout => 15
  }
}

Although I got stuck at the part where I need to add Deal Options to a Deal, is my current Logstash config correct? If it is, how can I complete it, thanks for your time!

Yousif
  • 527
  • 5
  • 23
Connectify
  • 23
  • 4

0 Answers0