Questions tagged [logstash-jdbc]
187 questions
0
votes
0 answers
Logstash bulk update documens in ElasticSearch with WHERE conditon
I have 2 tables in MySQL like this
Table DEPARTMENT
Id
Name
1
Department 1
2
Department 2
Table STAFF
Id
Department_Id
Name
1
1
Staff 1
2
1
Staff 2
3
2
Staff 3
4
1
Staff 4
STAFF table has about 10 million…
Ai Chau
- 23
- 4
0
votes
1 answer
Replace the last search result
I am currently monitoring the database(Jdbc).
This bank refers to product sales, where at first the status of the sale receives "WAITING FOR PAYMENT" and after payment it receives the status of "PAID".
however, when the payment status changes, the…
LMM
- 1
- 1
0
votes
0 answers
logstash elasticsearch output not deleting documents
I have the following config to get deleted entries from jdbc input and delete them from the ES index, I see that it runs and queries DB and gets entries but documents with those ids still exist in the index. What can be wrong here?
input {
jdbc {
…
Behlül
- 3,412
- 2
- 29
- 46
0
votes
0 answers
Elasticsearch array of an object using logstash
I have a mysql database working as a primary database and i'm ingesting data into elasticsearch from mysql using logstash. I have successfully indexed the users table into elasticsearch and it is working perfectly fine however, my users table has…
Hamza Khalid
- 21
- 3
0
votes
0 answers
LIMIT keyword is unknown by Teradata Database
I am using jdbc_static plugin with a simple select query
jdbc_static{
id => "JDBC_STATIC_APPLICATION_MIND_MAPPING"
loaders => [
{
id => "REMOTE_MAPPING"
query => "select field1, field2 FROM DB.view"
…
C.Mahfoud
- 90
- 2
- 9
0
votes
1 answer
Logstash pipeline pushing the data without transforming it to Elasticsearch
I'm using a docker-compose file with the ELK stack (Elastic, Logstash, Kibana). The docker-compose.yml file is very straight forward:
version: '3.8'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.3.2
…
ddieppa
- 5,866
- 7
- 30
- 40
0
votes
1 answer
How to switch multiple jdbc inputs in a single pipeline without type fields?
I know I can do something like this:
input {
jdbc {
type => "a"
}
jdbc {
type => "b"
}
}
output {
if [type] == "a" {
...
}
if [type] == "b" {
...
}
}
But as the offical document of jdbc input plugin said:
If you try…
Rick Dou
- 154
- 9
0
votes
1 answer
JDBC Logstash Elastic Kibana
I'm using JDBC input plugin to ingest data from mongodb to ElasticSearch.
My config is:
`input {
jdbc {
jdbc_driver_class => "mongodb.jdbc.MongoDriver"
jdbc_driver_library =>…
0
votes
1 answer
Have logstash JDBC input `last_run_metadata_path` on an external source?
We want to run logstash in an ECS environment where the instances have transient lives. They might be long lived, but they aren't permanent. Thus, having logstash store the sql_last_value in the last_run_metadata_path doesn't work if the instance is…
Chris Kessel
- 5,583
- 4
- 36
- 55
0
votes
1 answer
Connect through SSH tunnel on Logstash - Elastic cloud
I try to connect to my MySQL server with logstash on our elastic cloud cluster, the problem is that we use SSH tunnel on the sql server. Is there a way, using the logstash pipeline creation interface on elastic cloud, to connect to a mysql server…
FairPluto
- 697
- 6
- 28
0
votes
1 answer
how to push huge data in less time from Database table to Elastic search using log stash
I'm processing the 500 000 records from Postgres database to elastic using Logstash but it taking 40 minutes to completed the process. I want to reduce the process time and i have changed the pipeline.batch.size: 1000, pipeline.batch.delay: 50 in…
madhavi
- 13
- 5
0
votes
1 answer
Issue in ELK stack
i am using ELK stack 7.15.2
jdk-11.0.12
jre1.8.0_311
at C_3a_.ELK_20_Stack.logstash_minus_7_dot_15_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_input$1(C:/ELK…
Mian Muhammad Ramzan
- 223
- 2
- 8
0
votes
1 answer
logstash scheduler in yaml file
In the previous version of my logstash yaml file I used schedule => "* * * * *"
Now I'm trying to schedule it to tqice a day ( lets say 6 am and pm ), but it doesn`t work anymore.
File content:
input {
jdbc{
clean_run => true
…
Afik Menashe
- 77
- 1
- 1
- 8
0
votes
1 answer
Connection error from Oracle DB - Logstash
Has anyone faced this issue while connecting to an Oracle DB from Logstash JDBC Input Plugin :
[2021-07-22T11:22:42,912][ERROR][logstash.inputs.jdbc ][main][9615a4202c23f17db8abee168682c63c25349f105e2579d02086d38fe8145d97] Unable to connect to…
0
votes
1 answer
Why after set mapping, index return nothing?
I am using Elasticsearch 7.12.0 , Logstash 7.12.0, Kibana 7.12.0 on Windows 10 x64. Logstash config file logistics.conf
input {
jdbc {
jdbc_driver_library => "D:\\tools\\postgresql-42.2.16.jar"
jdbc_driver_class => "org.postgresql.Driver"
…
Vy Do
- 46,709
- 59
- 215
- 313