Stack Overflow
Stack Overflow

Questions tagged [logstash-jdbc]

187 questions
0
votes
0 answers

Reindexing/Updating Elasticsearch using Logstash on Jenkins

I would like to automate the process of updating the elasticsearch with latest data on demand and secondly, recreating the index along with feeding data using a Jenkins job. I am using jdbc input plugin for fetching data from 2 different databases…
user977815
  • 45
  • 3
  • 8
0
votes
0 answers

Parsing postgres date in logstash using datefilter plugin

I am using logstashto read the postgres data using jdbc input plugin and push to elastic The data is coming properly and things seems to be working fine just for a small problem i.e. My logs table has a field requesttimestamp with datatype of…
Vivek
  • 156
  • 3
  • 18
0
votes
1 answer

how logstash update same record in the elastic search with same id

I have a doubt on how the logstash updates the existing record in the elastic search index with the same document_id, and it executes as a scheduled job with a database (ES-LS-DB). Is logstash remove all the records in the index and re insert all or…
Anushka Ekanayake
  • 977
  • 2
  • 13
  • 33
0
votes
1 answer

undefined method `close_jdbc_connection' for #

I have been trying to import data from MS SQL Server to Elastic Search using Logstash. However, I am getting logstash pipeline and undefined method `close_jdbc_connection' error. I have not found the exact solution for this issue. The code used and…
Mrin
  • 13
  • 4
0
votes
0 answers

Creating an elasticsearch index from logstash

I am trying to load data from an Sql Server into ElasticSearch. I am using Logstash with the jdbc plugin and the elastic-search plugin. I am loading my data in ElasticSearch but can not figure out how to set my index. I am using a template index…
James
  • 493
  • 1
  • 10
  • 37
0
votes
0 answers

Logstash with queue enabled not ack http input events after jdbc input runs

I’m using logstash with queuing enabled. I’ve setup logstash to inject rows from mysql via the mysql input plugin on startup. Currently this is injecting 1846 rows. I also have a http input. When I take down ES and restart logstash as expected I get…
twiz911
  • 634
  • 1
  • 9
  • 18
0
votes
1 answer

Logstash jdbc-input-plugin configuration for initial sql_last_value

I synchronise data in Oracle database and ElasticSearch instance. Database table "SYNC_TABLE" has the following columns: "ID" which is a NUMBER, "LAST_MODIFICATION" - TIMESTAMP, "TEXT" - VARCHAR2. I use Logstash with jdbc-input-plugin in order to…
Ilya Zinkovich
  • 4,082
  • 4
  • 25
  • 43
0
votes
1 answer

Why is logstash merging tables?

I've got two configuration files pointing each on a different table of the database but in elasticsearch, i get the two tables merged so, tablea and tableb items will all be copy as typea and tablea and tableb items will all be copy as typeb. But i…
Jul6art
  • 219
  • 3
  • 14
0
votes
0 answers

How to handle markup (HTML) in logstash

I am using logstash with the JDBC driver to bulk import a bunch of data from SQL Server to Elasticsearch. (The end goal is to have this data be searchable from a web front-end.) One of the table columns contains HTML tags (,
anon
  • 4,578
  • 3
  • 35
  • 54
0
votes
2 answers

Logstash, mongodb and jdbc

I have a problem configuring logstash. I want to be able to put in input jdbc for mongodb. My config : input{ jdbc{ jdbc_driver_library => "mongo-java-driver-3.2.2.jar" jdbc_driver_class => "com.mongodb.MongoClient" …
Ad.Champred
  • 11
  • 2
0
votes
1 answer

Kibana Index Pattern showing wrong results

I am using ELK stack in which i have used jdbc input in logstash I have created 2 indexes users employees Both the indexes have one same column objid Logstash config file input { jdbc { jdbc_driver_library =>…
Harish Gupta
  • 364
  • 2
  • 8
  • 23
0
votes
1 answer

Logstash/Elasticsearch JDBC document_id vs document_type?

So im trying to wrap my head around the document_type vs document_id when using the JDBC importer from logstash and exporting to elasticsearch. I finally wrapped my head around indexes. But lets pretend im pulling from a table of sensor data (like…
msmith1114
  • 2,717
  • 3
  • 33
  • 84
0
votes
1 answer

Is it possible to use a varchar/string for jdbc sql_last_value?

I've already tested the sql_last_value with an integer and datetime type of fields in my jdbc plugin within the logstash input and it's working perfectly. I'm just curios, is there any possibility of using a varchar/string type of field, in order…
Kulasangar
  • 9,046
  • 5
  • 51
  • 82
0
votes
0 answers

Logstash jdbc input plugin gives date mapping error, while indexing data into elastic search from postgres

Background: I am trying to index data from Postgres into elastic search and following below steps: Step 1 :- Bulk import data from Postgres using logstash jdbc-input plugin. Step 2 :- Synchronize further metadata changes from application using…
ni3ns
  • 39
  • 7
0
votes
1 answer

Logstash - Syntax for a grok mutate gsub to replace backslashes by empty string

I have problem with combination of Oracle SQL statement and logstash' automatic escaping characters like \ and ". In SQL statement I have defined i.e.: to_char(milliseconds_to_date(m.originationtime),'yyyy-mm-dd\"T\"hh24:MI') But in elastic search…
Reddy SK
  • 1,334
  • 4
  • 19
  • 27
1 2 3
…
12
13