0

I am implementing solr 3.6 in my application.as i have the below data in my text file..

**

date=2011-07-08 time=10:55:06 timezone="IST" device_name="CR1000i" device_id=C010600504-TYGJD3 deployment_mode="Route" log_id=031006209001 log_type="Anti Virus" log_component="FTP" log_subtype="Clean" status="Denied" priority=Critical fw_rule_id="" user_name="hemant" virus="codevirus" FTP_URL="ftp.myftp.com" FTP_direction="download" filename="hemantresume.doc" file_size="550k" file_path="deepti/Shortcut to virus.lnk" ftpcommand="RETR" src_ip=10.103.6.100 dst_ip=10.103.6.66 protocol="TCP" src_port=2458 dst_port=21 dstdomain="myftp.cpm" sent_bytes=162 recv_bytes=45 message="An FTP download of File resume.doc of size 550k from server ftp.myftp.com could not be completed as file was infected with virus codevirus"

**

now i want to split above data based on key-value pairs..and want the each value to be indexed based on the key.. i want the changes should be in the configuraion files..i have gone through tokenizer in which whitespaceokenizer may work.but want the whole structure to be indexed..so can anyone please help me on this??? thanks..

Asha Koshti
  • 2,763
  • 4
  • 22
  • 30

2 Answers2

1

There is no tokenizer that I know of does this.

Using static fields:

You have to define all your "keys" as fields in schema.xml . They should have the relevant types (dates, string etc).

Create a POJO with these fields and parse this key/value pairs and populate the POJO. Add this pojo to solr using solrj.

Using dynamic fields:

In this case you dont need to define the keys in schema but use dynamic fields (based on the type of data). You still need to parse the key/value pairs and add to solr document. These fields need to be added using solrInputdoc.addField method.

As you define add new key/value pairs, the client would still need to know of the existence of this new key. But your indexer does not need to.

user1452132
  • 1,758
  • 11
  • 21
  • ..I have gone through with this link..stackoverflow.com/questions/3891054/… i want whenever the above line is given to solr..it should index on values based on field..just like whenever we give the XML. – Asha Koshti Jun 27 '12 at 06:03
  • 1
    This link's solution is for taking key=value and indexing terms key,value both as the given field. What you want seemed like you want only "values" indexed as solr fields defined as "keys" . – user1452132 Jun 27 '12 at 11:16
  • can i use PatternTokenizerFactory so only value is indexed not the key..but how will i map to that key??can you please guide? – Asha Koshti Jun 27 '12 at 11:41
  • Sorry, I could not think of a way to do this. It is probably easier to just pre-process the doc, split on "=" and only keep the vallues, and reassemble the document. – user1452132 Jun 27 '12 at 14:10
0

This cannot be done with a tokenizer. Tokenizers are called for each field, but you need processing before handing the data to a field.

A Transformer could probably do this, or you might do some straightforward conversion before submitting it as XML. It should not be hard to write something that reads that format and generates the proper XML format for Solr submissions. It sure wouldn't be hard in Python.

For this input:

date=2011-07-08 time=10:55:06 timezone="IST" device_name="CR1000i"

You would need to create the matching fields in a schema, and generate:

<doc>
  <field name="date">2011-07-08</field>
  <field name="time">2011-07-08</field>
  <field name="timezone">IST</field>
  <field name="device_name">CR1000i</field>
  ...

Also in this pre-processing, you almost certainly want to convert the first three fields into a single datetime in UTC.

For details about the Solr XML update format, see: http://wiki.apache.org/solr/UpdateXmlMessages

The Apache wiki is down at this exact moment, so try again if there is an error page.

Walter Underwood
  • 1,201
  • 9
  • 11
  • ..I have gone through with this link..http://stackoverflow.com/questions/3891054/how-can-i-set-up-solr-to-tokenize-on-whitespace-and-punctuation..now i want whenever the above line is given to solr..it should index on values based on field..just like whenever we give the XML.. – Asha Koshti Jun 27 '12 at 05:55
  • That is not possible with a tokenizer. Do processing outside of Solr or write a transformer, as I said in the answer. – Walter Underwood Jun 27 '12 at 15:52
  • ..how would i will write transformer and do indexing with it..can you elaborate or provide me some links..?? – Asha Koshti Jun 29 '12 at 05:27
  • Actually, I think it should be an Entity Processor. This is a part of Solr that I have not used, but you can start here: http://wiki.apache.org/solr/DataImportHandler#EntityProcessor – Walter Underwood Jul 10 '12 at 19:39