Tika fetches the binary content stored in database but does not indexes it

Question

I am trying to parse the binary content data stored in database in table document_attachment in column file_data and trying to index the same so that it's content becomes available for searching using Solr. When I run the indexer it fetches the rows which is twice in number to the rows returned by running the query in entity named "dcs" and throws no errors or exceptions. it however does not indexes the binary content(the field that I associate with tika despite of fetching it from the table).

I am using apache-solr-3.6.1 and Tika 1.0

My configuration files look something like :

data-config.xml

<?xml version="1.0" encoding="UTF-8"?>
<dataConfig>
 <dataSource
          driver="com.mysql.jdbc.Driver"
          url="jdbc:mysql://localhost/espritkm_1?zeroDateTimeBehavior=convertToNull"
          user="root"
          password=""
          autoCommit="true" batchSize="-1"
          convertType="false"
          name="test"
          />

  <dataSource name="fieldReader" type="FieldStreamDataSource" />
  <document name="items">
  <entity name="dcs"
          query="SELECT 222000000000000000+d.id AS common_id_attr,d.id AS id,UNIX_TIMESTAMP(d.created_at)  AS date_added,d.file_name as common1, d.description as common2, d.file_mime_type as common3, 72 as common4,(Select group_concat(trim(tags) ORDER BY trim(tags) SEPARATOR ' | ') from tags t where t.type_id = 72 AND t.feature_id = d.id group by t.feature_id) as common5,d.created_by as common6, df.name as common7,CONCAT(d.file_name,'.',d.file_mime_type) as common8,'' as common9,(Select da.file_data from document_attachment da where da.document_id = d.id) as text  FROM document d LEFT JOIN document_folder df ON df.id = d.document_folder_id  WHERE d.is_deleted = 0 and d.parent_id = 0 " dataSource="test" transformer="TemplateTransformer">

<field column="common_id_attr" name="common_id_attr" />
    <field column="id" name="id" />
        <entity dataSource="fieldReader" processor="TikaEntityProcessor" dataField="dcs.text" format="text" pk="dcs.id" >  

    <field column="text" name="text" />
   </entity>
 </entity>

schema.xml

   <schema>
    <fields> 
     <field name="common_id_attr" type="string" indexed="true" stored="true" multiValued="false"/>
     <field name="id" type="string" indexed="true" stored="true"/>
     <field name="text" type="text" indexed="true" stored="true" multiValued="true"/>
   </fields> 
  <uniqueKey>common_id_attr</uniqueKey>
  <solrQueryParser defaultOperator="OR"/>
  <defaultSearchField>text</defaultSearchField>
 </schema>

Though the rows it fetches is double the number of documents counting the rows of tika as separate (I don't understand why?). It does not indexes binary content.

I am stuck in this problem from long. Can someone please help

What do you have in the field _text_ after db import is done? No values at all? — Alexey Serba, Feb 04 '13 at 14:21

score 1 · Accepted Answer · answered Feb 04 '13 at 13:08

I was able to index the documents using Apache Solr version 3.6.2. I have described the steps here:

http://tuxdna.wordpress.com/2013/02/04/indexing-the-documents-stored-in-a-database-using-apache-solr-and-apache-tika/

I think it should be doable in 3.6.1 as well. I was only impatient to search for a tarball of version 3.6.1 when only 3.6.2 was avaiable from the official site.

I hope that helps.

Tika fetches the binary content stored in database but does not indexes it

1 Answers1