3

The android [FTS4 documentation][1] for android states "This class will have a mapping SQLite FTS4 table in the database". Should I interpret that as a mapping SQLite FTS4 table will be generated simply by adding the annotation, or does it mean I need to provide some sort of mapping table to 'wire up' the FTS4 entity class to a virtual FTS4 table?

I'm currently assuming the former. If that's correct, then how can I make use of this mapping table in practical terms? I think it's right to say that when you create an entity with the FTS4 annotation it generates an ordinary table and a mapping table, neither of which is actually a virtual FTS4 table which can be queried with the MATCH function. So what exactly does the the mapping table map to?! Some hypothetical virtual FTS4 external content table that's implemented at runtime, possibly during a database callback or migration? In that case how am I supposed to reference the searchable FTS4 table in my DAO without getting a compiler error?

SOLVED:

Turns out the former interpretation was correct and that the mapping table generated includes the virtual table required for a full text search. I do still think the documentation could be clearer on this, but in the end the real underlying issue was that my room dependency wasn't up to date. D'oh!

Josh Hardman
  • 721
  • 6
  • 17

1 Answers1

9

With FTS 3 or 4 you define the core table and then the VIRTUAL table with the USING keyword and the appropriate parameters (module such as FTS3 or FTS4 and the parameters that the module expects). The FTS module then generates the tables.

E.g. A database has a standard bible table, with columns book TEXT, chapter INTEGER, verse INTEGER and content TEXT. For FTS you create the virtual table using something like

CREATE VIRTUAL TABLE bible_fts USING fts3(book,chapter INTEGER, verse INTEGER, content TEXT)
  • this is the only special table as such, the following tables are normal tables other than that the FTS module creates them.

When the CREATE statement is executed then table bible_fts_content, bible_fts_segments and the bible_fts_segdir are created. (may be a little different for FTS4). The FTS ready version of SQLite will then be able to handle things such as MATCH.

In short, it's all dependant upon the special handling of the defined module.

In case you couldn't tell, it's very unclear to me what my responsibilties are here!

My understanding is that you have to define the core tables.

You code the relevant @FTS (3 or 4) annontation Room generates the vritual table e.g. you get something along the lines of :-

    _db.execSQL("CREATE TABLE IF NOT EXISTS `device_item` (`id` INTEGER, `initial` TEXT, `added1` INTEGER NOT NULL DEFAULT 0, `added2` TEXT DEFAULT '', PRIMARY KEY(`id`))");
    _db.execSQL("CREATE VIRTUAL TABLE IF NOT EXISTS `table1` USING FTS3(`name` TEXT, `mapToTable2` INTEGER NOT NULL, `mapToTable3` INTEGER NOT NULL)");
    _db.execSQL("CREATE TABLE IF NOT EXISTS `table2` (`id` INTEGER, `nameOfT2` TEXT, `anotherNameOfT2` TEXT, PRIMARY KEY(`id`))");
    _db.execSQL("CREATE TABLE IF NOT EXISTS `table3` (`id` INTEGER, `nameOfT3` TEXT, `anotherNameOfT3` TEXT, PRIMARY KEY(`id`))");
    _db.execSQL("CREATE TABLE IF NOT EXISTS room_master_table (id INTEGER PRIMARY KEY,identity_hash TEXT)");
    _db.execSQL("INSERT OR REPLACE INTO room_master_table (id,identity_hash) VALUES(42, '91a23aea1ab6e684828fad82668cb9a5')");
  • P.S. with Room the generated code is virtually the room bible.
  • The above wouldn't work because I just put @FTS3 and any old table (there are restrictions). I just did it to demo.

Then I believe that because of the annotations that Dao's etc reference what they need. However, I suspect that you will have to be coding @Ignore frequently. (I've never actually used FTS, my experience was with a tool that I was playing with a tool convert non-room databases to room and got the conversion working with FTS i.e. to the stage where Room accepted the database schema wise).

You may also find Enabling SQLite FTS in Room 2.1 of use.

Example

The core table

Entity Bible.java

@Entity(tableName = "bible")
public class Bible {

    @PrimaryKey
    Long id;
    String book;
    Long chapter;
    Long verse;
    String content;

    public Bible(){}

    @Ignore
    public Bible(String bookName, Long chapter, Long verse, String content) {
        this.book = bookName;
        this.chapter = chapter;
        this.verse = verse;
        this.content = content;
    }
    // ...... getters and setters nothing special
}

The FTS Entity

As book and content are the textual columns these will be defined for FTS

BibleFTS.java

@Fts4(contentEntity = Bible.class) //<<<<<<<<<< the table to FTSalise
@Entity(tableName = "bible_fts")
public class BibleFTS {

    @PrimaryKey
    Long rowid; //<<<<<<<<<< MUST HAVE 
    String book;
    String content;
}
  • Unsure about the need for getters and setters, first attempt worked without
  • Unsure about the affect autogenerate (but Long doesn't need @NonNull not autogenerate)

Dao

BibleDao.java (covers Bible and BibleFTS)

@Dao
public interface BibleDao {

    @Insert
    Long insertBibleRow(Bible bible);

    @Insert
    Long[] insertBibleRows(Bible... bibles);

    @Query("SELECT * FROM bible")
    List<Bible> getAllBibleRows();

    //<<<<<<<<<< USED TO REBUILD (Room might do this anyway think I saw it somewhere) >>>>>>>>>>
    @Query("INSERT INTO bible_fts(bible_fts) VALUES('rebuild')")
    void rebuild();

    //<<<<<<<<<< SEARCH The searchable columns >>>>>>>>>
    @Query("SELECT bible.book,bible.chapter,bible.verse,bible.content FROM bible_fts JOIN bible ON id = docid WHERE bible_fts MATCH :searchFor")
    List<Bible> searchFor(String searchFor);

}

Database

Database.java

@androidx.room.Database(
        version = 1,
        entities = {
                Bible.class, //<<<<<<<<<<
                BibleFTS.class //<<<<<<<<<<
        })
abstract class Database extends RoomDatabase {


    abstract BibleDao bibleDao(); //<<<<<<<<<<
}

Test Activity

MainActivity.java (onCreate method)

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);

    mDatabase =  Room.databaseBuilder(this,Database.class,"devicitems")
            .allowMainThreadQueries()
            .build();
    mBibleDao = mDatabase.bibleDao();
    Bible b = new Bible("Geneisis",1L,1L,"In the begining some bright spark made a thinngy called Earth and said I shall rule over that little speck.");
    mBibleDao.insertBibleRow(b);
    mBibleDao.rebuild();
    List<Bible> found = mBibleDao.searchFor("in");
    for (Bible currentBible: found) {
        Log.d(
                "FOUNDIN",
                "Found in in :- Book" + currentBible.getBook() +
                        " - Chapter " + currentBible.getChapter() +
                        " - Verse " + currentBible.getVerse() +
                        "\n\t" + currentBible.getContent()
        );
    }
}

Result

2019-10-12 21:37:00.584 30255-30255/? D/FOUNDIN: Found in in :- BookGeneisis - Chapter 1 - Verse 1
        In the begining some bright spark made a thinngy called Earth and said I shall rule over that little speck.
MikeT
  • 51,415
  • 16
  • 49
  • 68
  • Thanks for the links Mike. I'm aware of the SQL requirements for creating a virtual table for FTS4 but I'm having a bit of trouble figuring out where it fits in the android Room architecture pattern. It's unclear whether the virtual table is generated behind the scenes as a result of annotating the entity class. If it's not, I'm not sure where I can create the virtual table so that I'd be able to get a reference to it for the SQL query in the DAO class. I've tried to implement it as a migration, as a callback and with the initial database but no glory so far. – Josh Hardman Oct 12 '19 at 09:04
  • I've just watched a video from the Android dev summit which seems to be saying that the annotation automatically generates the virtual table. However, I just tried implementing it in the way they suggested and I'm getting an "unable to use function MATCH in the requested context (code 1)" error. This suggests that the table generated by the FTS4 Room Entity class might not actually be the virtual table required for the query. Either that or there's some other issue I'm unaware of! – Josh Hardman Oct 12 '19 at 09:05
  • [Here's the video](https://www.youtube.com/watch?v=sU-ot_Oz3AE) – Josh Hardman Oct 12 '19 at 09:06
  • I've been referring to that Medium article a fair bit, but some of it doesn't seem to add up! For example the CREATE VIRTUAL TABLE IF NOT EXISTS statement won't create a new table as the name of the table is the same as the table produced by the entity which I really suspect isn't actually a virtual table. The reason i say that is when I tried to run the next line of SQL ( table_name(table_name) VALUES ('rebuild')) command it says that the column doesn't exist. A virtual table should have a hidden column with its own name as the column name but in this case it doesn't! – Josh Hardman Oct 12 '19 at 09:24
  • Also the entity shown doesn't have a Primary Key defined which means it won't compile without an error. And the SQL query appears to have the virtual table and ordinary table the wrong way round!! From the android dev presentation and a few stack overflow answers it appears that the virtual table should be queried first – Josh Hardman Oct 12 '19 at 09:27
  • I've tried your working example but I'm still getting compiler errors which say the docid and self-referential columns don't exist ( `There is a problem with the query: [SQLITE_ERROR] SQL error or missing database (no such column: docid)` ). It seems like the table reference from the fts4 entity class isn't actually referring to a virtual table. I'm writing in Kotlin not Java so it might be something to do with the Kotlin compiler (the error is a Kotlin compiler error), or perhaps some annotation I've unwittingly omitted which is required to compile the table properly in Kotlin. – Josh Hardman Oct 13 '19 at 00:20
  • If it's the latter, I've yet to come across any documentation or tutorials which mention that. – Josh Hardman Oct 13 '19 at 00:20
  • Also when I try to run the search using references to the columns defined in the Entity (rowid // values column) I get the following compiler error `exception: unable to use function MATCH in the requested context (code 1)` – Josh Hardman Oct 13 '19 at 00:44
  • 1
    Have you got the kapt plugin in your build gradle? e.g. `apply plugin: 'kotlin-kapt'` along with `kapt 'androidx.room:room-compiler:2.2.0-rc01'` – MikeT Oct 13 '19 at 00:53
  • Yes I had but I've just figured out what was causing the issue this whole time. I'd updated my room dependency a while ago to 2.1.0 but just noticed that it'd slipped back to 2.0.0. I must have ended a session without saving and not realised. I've updated it again and the search appears to be functional now. As always, the devil's in the detail! Thanks for all your help Mike!! – Josh Hardman Oct 13 '19 at 01:24