I have 2 tables in MySQL like this
Table DEPARTMENT
Id | Name |
---|---|
1 | Department 1 |
2 | Department 2 |
Table STAFF
Id | Department_Id | Name |
---|---|---|
1 | 1 | Staff 1 |
2 | 1 | Staff 2 |
3 | 2 | Staff 3 |
4 | 1 | Staff 4 |
STAFF
table has about 10 million records.
All STAFF
's informations has been pushed by Logstash to ElasticSearch. Each document in ElasticSearch now only have 3 fields are Staff_Id
, Staff_Name
and Department_Name
. Something like this:
{
"Staff_Id": 1,
"Staff_Name": "Staff 1",
"Department_Name": "Department 1"
}
Because of practical needs, I need to add one more field called Department_Id
to each document. Note that this field (Department_Id
) does not exist on existing documents.
I am a newbie to both Logstash and ElasticSearch. How can I do this with Logstash? Interpreted in the SQL way would be:
- SELECT * FROM DEPARTMENT;
- UPDATE STAFF SET Department_Id = XXX WHERE Department_Name = YYY
Note that DEPARTMENT
table has about 100.000 records and ElasticSearch has about 10 million documents.
Can you take a look?