1

I am pretty new to Solr and requires some help. Currently I am pushing all my logs to be indexed by Solr using Flume-ng and Syslog-ng. I would also like to index the date, but I keep on getting exception due to different time format.

Syslog-ng generates ISO date/time according to this format: 2013-12-24T10:36:24.0000+8:00 or in general YYYY-MM-DDTHH:MM:SS:ffffTZD but Solr is only accepting in Zulu time YYYY-MM-DDTHH:MM:SSZ.

  1. Is there any way for me to create a custom Solr date parser so I can correctly parse the date/time. Google gave me no result of how to approach this.

  2. Or is there any way syslog-ng can generates the date/time in format accepted by Solr?

Currently, as a workaround I had to set the servers to UTC and generate the date/time manually using the template. However, I would prefer not to set all servers to UTC since it can be in different timezone and I would like to keep it that way.

Shulhi Sapli
  • 2,286
  • 3
  • 21
  • 31
  • Would you give more detail about the import process? How are you getting the data into Solr? There are several points to hook in. – cheffe Dec 30 '13 at 10:01
  • The data will be imported from HBase. Flume-ng will save the data into HDFS in which I will run a batch map/reduce and the outputs are saved into HBase. Then it will be indexed by Solr. – Shulhi Sapli Dec 30 '13 at 10:28
  • So Solr is pulling the data via a [DataImportHandler like this one](http://code.google.com/p/hbase-solr-dataimport/wiki/ConfigurationExample)? – cheffe Dec 30 '13 at 10:46
  • No, I'm not using `DataImportHandler` if I understand this correctly. I'm actually using HBase Indexer (Lily Hbase Indexer) so whenever my Hbase table is updated, it will also trigger the indexer. FYI, I'm deploying hadoop cluster using CDH. Let me know if you need more detail. Thanks again for helping. – Shulhi Sapli Dec 30 '13 at 11:59

0 Answers0