0

I have 2 tables in MySQL like this

Table DEPARTMENT

Id Name
1 Department 1
2 Department 2

Table STAFF

Id Department_Id Name
1 1 Staff 1
2 1 Staff 2
3 2 Staff 3
4 1 Staff 4
  • STAFF table has about 10 million records.

All STAFF's informations has been pushed by Logstash to ElasticSearch. Each document in ElasticSearch now only have 3 fields are Staff_Id, Staff_Name and Department_Name. Something like this:

{
    "Staff_Id": 1,
    "Staff_Name": "Staff 1",
    "Department_Name": "Department 1"
}

Because of practical needs, I need to add one more field called Department_Id to each document. Note that this field (Department_Id) does not exist on existing documents.

I am a newbie to both Logstash and ElasticSearch. How can I do this with Logstash? Interpreted in the SQL way would be:

  • SELECT * FROM DEPARTMENT;
  • UPDATE STAFF SET Department_Id = XXX WHERE Department_Name = YYY

Note that DEPARTMENT table has about 100.000 records and ElasticSearch has about 10 million documents. Can you take a look?

Ai Chau
  • 23
  • 4
  • can you provide your logstash configuration ? also, while indexing document from logstash, did you provided custom doc_id or it is auto generated ? – Sagar Patel Oct 31 '22 at 04:42
  • Does this answer your question? [Does Logstash support Elasticsearch's \_update\_by\_query?](https://stackoverflow.com/questions/53330232/does-logstash-support-elasticsearchs-update-by-query) – Paulo Oct 31 '22 at 10:50

0 Answers0