Questions tagged [logstash-jdbc]
187 questions
1
vote
0 answers
Logstash jdbc input with postgresql - jdbc_fetch_size ignored?
I am ingesting a large amount of data from postgresql and am trying to get jdbc_fetch_size working properly. Before this I used redshift and it was working very well. Logstash memory would stay around 2 to 3gb. I switched to postgresql and memory is…
Casey
- 2,611
- 6
- 34
- 60
1
vote
0 answers
Logstash filter: aggregate nested arrays
I'm trying to fetch data from MySQL and push it to ElasticSearch using Logstash and I'm having trouble creating a config file for Logstash that suits my need
I'm trying to achieve this result:
{
"products":[
{
"id":1,
…
Connectify
- 23
- 4
1
vote
0 answers
Logstash is retaining messages for few hours in Kubernetes
I'm trying to use logstash Jdbc plugin to synchronize data from a Postgres database to some ouput (elastic, rabbitMq).
The problem is that logstash is running well every hour (I configured the cronJob like that).
Logstash is successfully reading the…
Omar
- 11
- 1
1
vote
1 answer
How sync mongodb collection with elasticseatch index using logstash?
I'm tring to use logstash-7.6.2 to sync my mongo with elasticsearch. I'm using dbschema jdbc driver.
input {
jdbc{
jdbc_driver_class => "com.dbschema.MongoJdbcDriver"
jdbc_driver_library =>…
Anton Magov
- 186
- 1
- 14
1
vote
1 answer
Logstash Input -> JDBC in some properties or parameterizable file?
I am using logstash to ingest elasticsearch. I am using input jdbc, and I am urged by the need to parameterize the inputt jdbc settings, such as the connection string, pass, etc, since I have 10 .conf files where each one has 30 jdbc and 30 output…
Max
- 538
- 1
- 6
- 16
1
vote
1 answer
Remove input tags field with elasticsearch and logstash
I'm using logstash to send to elasticsearch, would someone know how to remove the [tags] field?
I am using this field to filter where each jdbc input should enter, I leave an example below.
What I would like is for that field not to be inserted into…
Max
- 538
- 1
- 6
- 16
1
vote
2 answers
SQL Server database > Logstash > Elasticsearch: map result set records related to the same entity to same ES document
I have question and answers entities represented in a SQL Server database as 2 tables Questions and Answers (see below). The relationship between them is OneToMany.
Questions table
Id Title
-------------------
1 Question 1
2 …
Hutsul
- 1,535
- 4
- 31
- 51
1
vote
1 answer
JDBC not working for logstash and mongoDB connection
This Configuration gives the output as [unable to connect to the database and no suitable driver found!], i am using logstash version 7.5.1
SanTiaGo KarThiCk
- 33
- 2
1
vote
0 answers
Delete data or document from elastic search using logstash
I am trying to delete elastic search data or document using logstash configuration but delete seems to be not working.
I am using logstash 5.6.8 version
Below is the logstash configuration file:
```input {
jdbc {
#db configuration
'''
…
user10566909
- 43
- 8
1
vote
2 answers
logstash-input-jdbc how to use utf-8 chars in statement
I use logstash-input-jdbc to sync my database to elasticsearch.
Env: (logstash 7.5, elasticsearch 7.5,mysql-connector-java-5.1.48.jar, logstash-input-jdbc-4.3.16)
materials.conf:
input {
jdbc {
jdbc_connection_string =>…
LF00
- 27,015
- 29
- 156
- 295
1
vote
0 answers
Cannot aggregate data from jdbc to output one single document
I'm trying to ingest data from jdbc into elastic search via logstash but i think the aggregate function is not working
my pipeline code is
input {
jdbc {
jdbc_driver_library => "/home/simrat/Downloads/mysql-connector-java-5.1.6.jar"
…
Simrat Khanuja
- 13
- 4
1
vote
1 answer
Does Logstash maintain a registry file?
I am using Logstash to send data to Elasticsearch using jdbc as input which takes data from MySQL table. Everything is working fine, but I have an issue that whenever I delete the index I don't get the same data back which was in that index.
Does…
Aman Prakash
- 340
- 4
- 15
1
vote
1 answer
ERROR on fetching data from mysql db using jdbc logstash plugin
When I try to fetch data from mysql db using jdbc logstash plugin, I'm facing below error
[2019-07-10T07:34:31,806][ERROR][logstash.javapipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#
Amit
- 61
- 3
1
vote
1 answer
jdbc_static: Getting a sql syntax error I did'nt write
With logstash I am trying to Extract some tables, Transform them locally on the logstash mashine, and then Load the result to ElasticSearch. The reason for this solution is due to very limited computing power on the source server, a MariaDB.
I have…
Erik Warming
- 184
- 1
- 2
- 14
1
vote
0 answers
How to use mongo db plugin for logstash?
I need to push data to the elastic search from mongo db. There is no offcial plugin available for mongodb. the, how can I import data from mongodb and push to elastic search.
I try to use this GIT
but, i cant install the plugin. I am getting error…
Hariharan AR
- 1,386
- 11
- 20