16

I'm new to Solr and am looking for a way to record searches (or keywords) to a log file or database so that I can then analyse for data visualisation.

  • Can Solr do this already?
  • Is this data accessible via. a Solr query?

Thanks.

Update 1

I'm starting to think I might need to write my own Solr analyzer?

Community
  • 1
  • 1
Ryall
  • 12,010
  • 11
  • 53
  • 77

4 Answers4

11

I think it depends on what you are looking to log? Are you just looking to record the queries users are submitting as well as the results? If it's just "what are folks searching for" then you have that data in the q parameter that is logged by the servlet container. If you are using the default Jetty setup, look at ./logs/*request.log. You will see lines like:

0:0:0:0:0:0:0:1%0 -  -  [21/01/2010:15:08:29 +0000] "GET /solr/select/?q=*:*&qt=geo&lat=45&long=15&radius=10 HTTP/1.1" 200 197

In this case, you can parse out that the user was doing a q=: search! Use a tool like AWStats to parse your logs and do the analysis. It's at least a quick and easy way to get some information!

Eric Pugh
  • 1,563
  • 11
  • 17
  • This would be very quick to write a script for, if one is not inclined to write java code (wrt @yuval answer). – memnoch_proxy Jan 21 '10 at 19:21
  • What are the last two numbers (Here: '200 197') of the log entry? - Thx – RngTng Aug 29 '11 at 15:26
  • RngTng, the numbers is the response code, HTTP 200, which means all okay, and 197 is the time (I think) in milliseconds. Might be size of results as well, not sure. – Eric Pugh Jan 10 '12 at 16:59
  • @EricPugh Found the request.log but that file is empty. Please advise. – sunskin Dec 30 '13 at 17:24
4

Months later ... maybe someone is interested:

http://karussell.wordpress.com/2010/10/27/feeding-solr-with-its-own-logs/

(you'll need to adapt the log parser if you are not using the default solr output format)

Karussell
  • 17,085
  • 16
  • 97
  • 197
2

The SolrLogging wiki page says you can use JDK logging (in Solr 1.0 to 1.3) or slf4j logging in Solr 1.4. About your own Solr analyzer - it depends on your needs. In many cases, using your own analyzer helps for specific retrieval requirements.

Yuval F
  • 20,565
  • 5
  • 44
  • 69
  • This would be very appropriate because you might want to use jdbc to record those findings. Depending on the ferocity of the traffic, making a synchronous jdbc call during a search might be a bottleneck, though. Might be better to log to a file and parse in a separate process. – memnoch_proxy Jan 21 '10 at 19:23
2

You can look at something like logstash to parse your log data.

jsp
  • 2,546
  • 5
  • 36
  • 63
  • Here is a good write up http://blog.comperiosearch.com/blog/2015/09/21/solr-logstash-analysis/ – cheffe Jul 27 '17 at 07:41