Solr dataimport change dataSource dynamically

Question

I have done the following settings for dataimport from about 20 mdb files using ucanaccess:

<?xml version="1.0" encoding="UTF-8" ?>
<dataConfig>
<dataSource name="a" driver="net.ucanaccess.jdbc.UcanaccessDriver" type="JdbcDataSource" url="jdbc:ucanaccess://E:/feqh/main.mdb;memory=false" />

<dataSource name="a1" driver="net.ucanaccess.jdbc.UcanaccessDriver" type="JdbcDataSource" url="jdbc:ucanaccess://E:/feqh/A/1.mdb;memory=false" />
<dataSource name="a2" driver="net.ucanaccess.jdbc.UcanaccessDriver" type="JdbcDataSource" url="jdbc:ucanaccess://E:/feqh/A/2.mdb;memory=false" />
<dataSource name="a3" driver="net.ucanaccess.jdbc.UcanaccessDriver" type="JdbcDataSource" url="jdbc:ucanaccess://E:/feqh/A/3.mdb;memory=false" />
<dataSource name="a4" driver="net.ucanaccess.jdbc.UcanaccessDriver" type="JdbcDataSource" url="jdbc:ucanaccess://E:/feqh/A/4.mdb;memory=false" />
<!-- and so on -->

<document>
    <entity name="Book" dataSource="a"
            query="select bkid AS id, bkid AS BookID,bk AS BookTitle, betaka AS BookInfo, cat as cat from 0bok">
                   <field column="id" name="id"/>
                   <field column="BookID" name="BookID"/>
                   <field column="BookTitle" name="BookTitle"/>
                   <field column="cat" name="cat"/>    
        <entity name="Category" dataSource="a"
                query="select name as CatName, catord as CatWeight, Lvl as CatLevel from 0cat where id = ${Book.CAT}">
                    <field column="CatName" name="CatName"/>
                    <field column="CatWeight" name="CatWeight"/>
                    <field column="CatLevel" name="CatLevel"/>
        </entity>

        <entity name="Pages" dataSource="a5" onError="continue"
                query="SELECT nass AS PageContent, page AS pageNum FROM b${Book.ID} ORDER BY page">
                <field column="PageContent" name="PageContent" />
                <field column="PageNum" name="PageNum" />
                <entity name="Titles" dataSource="a5" onError="continue"
                    query="SELECT * FROM t${Book.ID} WHERE id = ${Pages.PAGE} ORDER BY sub">
                        <field column="ID" name="TitleID"/>
                        <field column="TIT" name="PageTitle"/>
                        <field column="SUB" name="TitleWeight"/>
                        <field column="LVL" name="TitleLevel"/>
                </entity>
        </entity>


    </entity>
  </document>
</dataConfig>

In every time I liked to import from a different dataSource I had to change dataSource attribute manually for both Pages and Titles entities, then perform dataimport without clean. Now with more than 600 mdb files, it is not an wise option. Is there any way to make looping inside the config? In other words: there is a main entity or mdb files that handles all books titles and categories then every book has its own mdb file named with its id for example 245.mdb for the book of id 245, So I need to change the dataSource for Pages and Titles dynamically.

Seems to me you should have a single .mdb with book, category and page tables therein for all your data. Having separate file for each book's pages does not scale as you add more books. Also, can you move all of this data into something other than Microsoft Access? — orangepips, Oct 03 '14 at 17:01
@orangepips I, already, tried to search for any tool that convert the MS Access database to MySQL but it is the same idea, I have to manipulate more than 600 mdb file manually! Currently, if I have not got a direct Solr solution, I plan to make a PHP application that automate the process, but I will need to know how to make non-clean dataimport via HTTP. — SaidbakR, Oct 03 '14 at 17:38

score 2 · Answer 1 · answered Oct 04 '14 at 04:06

You cannot create dataSources in a loop, but I believe you can pass dataSource information in a parameter variable. So, perhaps, you can loop over your collection outside of Solr and then trigger DIH with the correct source as a parameter variable.

Just ensure to run DIH in sync mode to avoid different calls stepping on each other (I think the param is syncMode)

Solr dataimport change dataSource dynamically

1 Answers1