How to speed up Android app loading 140,000-item ArrayList from an Asset

Question

I have a Java program that I just turned into a slooooooow-to-load Android app. The problem: it deals with a 140,000-word "dictionary" (stored in an Asset file) in which to look for words matching "Windows wildcard" patterns: E.g., S???CK* would match STICKS, SHACK, STACK, , STACKOVERFLOW, etc. It's WAY fast in Windows 7. Not so on phone.

One thing I've done is to read all 140,000 words into an ArrayList (I was shocked that it compiled and ran), after which, as long as the pattern doesn't begin with a wildcard, Collections.binarySearch(...) makes lookup virtually immediate.

But reading it into the array list takes 60 seconds and user input is blocked. And it happens every time onCreate has to be run--i.e., unacceptably often.

I want to speed that up.

Here's an SSCCE of what works perfectly but far too slowly:

MainActivity.java

public class MainActivity extends Activity {

  @Override
  protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);

    FragmentTransaction
        ft;
        ft = getFragmentManager().beginTransaction();
        ft.replace(R.id.layout_container, new OutputFragment());
        ft.commit();
  };
}

OutputFragment.java

public class          OutputFragment          extends Fragment
{
  @Override
  public void onCreate(Bundle savedInstanceState)
  {
    super.onCreate(savedInstanceState);
  }

  @Override
  public View onCreateView(LayoutInflater _layoutInflater,
                           ViewGroup      _sourceOfLayoutParams,
                           Bundle         savedInstanceState)
  {


    View v = _layoutInflater.inflate(R.layout.fragment_output,_sourceOfLayoutParams, false);

    EditText et = (EditText)v.findViewById(R.id.txaOutput);

    Matcher matcher = new Matcher(getActivity().getAssets());

    for (int i = 0; i < 9; i++)
       et.append("\n" + matcher.get(i));

   return v;
  }
} 

Matcher.java

public class Matcher extends ArrayList<String> {

  Matcher(AssetManager assets) {

    Scanner scDict = null;
    try { scDict = new Scanner(assets.open("dictionary.dic")); }
    catch (IOException e) { e.printStackTrace(); }

    int k = 0;

    while(scDict.hasNext())// && ++k<10)
      add(scDict.next());
  }
}

activity_main.xml

<RelativeLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
          android:layout_width    ="match_parent"
          android:layout_height   ="match_parent"

    xmlns:tools  ="http://schemas.android.com/tools"
          tools:context           =".MainActivity"
    >

   <LinearLayout
       android:id           ="@+id/layout_container"
       android:orientation  ="vertical"
       android:layout_width ="match_parent"
       android:layout_height="match_parent">
   </LinearLayout>

</RelativeLayout>

fragment_output.xml

<GridLayout
      xmlns:android="http://schemas.android.com/apk/res/android"
            android:layout_width ="match_parent"
            android:layout_height="match_parent"
      android:rowCount="33"
      android:columnCount="2">

    <TextView
        android:id              ="@+id/txvOutput"
        android:text            ="Output shown below"
        android:layout_width    ="wrap_content"
        android:layout_height   ="wrap_content"
        android:textAppearance  ="?android:attr/textAppearanceLarge"
        android:layout_row="0"
        android:layout_column="0">
    </TextView>

        <EditText
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:text="New Text"
            android:id="@+id/txaOutput"
            android:layout_row="2"
            android:inputType="textMultiLine"
            android:layout_column="0"
            android:maxLines="100"/>


</GridLayout>

So what I want is to speed it up. I have read "Keeping Your Android App Responsive". I don't know if I can make it fit my situation. I took the example from there and adjusted it as best I could:

  private class LoadWords extends AsyncTask<Scanner, Integer, Long> 
  {
    @Override
    protected Long doInBackground(Scanner... params) { // variable arg list required (??)
      while(params[0].hasNext()) //// no way this could work...
        add(params[0].next());
      return 0L;
    };
  }

I didn't expect it to work as soon as I typed params[0].hasNext()but a variable argument list seems to be required.

Here's how I tried to implement it:

    LoadWords loadWords = new LoadWords(); /////////////////////////

    InputStream stream = null;
    Scanner     scDict = null;
    ...
                stream = assets.open("dictionary.dic");
    ...    

    scDict = new Scanner(stream);

    loadWords.execute(scDict); ///////////////////// What should I pass?????

I imagine I should give up this approach and try to use a Thread that I will have to manage. I'm not comfortable with this.

Any suggestions about how to proceed are welcome.

Answer 1

I have a partial solution, using AsyncTask. Loading the 140,000 words occurs in background; GUI is responsive immediately, BUT not all words loaded in time for returning all matches.

public class ListMaker extends ArrayList<String> 
{
  Scanner scDict;
  InputStream stream = null;

  public Matcher(AssetManager assets) 
  {
    LoadWords loadWords = new LoadWords();
    stream = assets.open("dictionary.dic");
    loadWords.execute((Object[]) null);
  }

  private class LoadWords extends AsyncTask<Object, Integer, ArrayList<String>> {
    @Override
    protected ArrayList<String> doInBackground(Object... params) 
    {
      scDict = new Scanner(stream).useDelimiter("\r\n");
      while (scDict.hasNext())
        add(scDict.next());
      return null;
    }

    @Override
    protected void onPostExecute(ArrayList<String> result) {
      MainActivity.setLoaded(true);
    }

Answer 2

Because of Paulo Avelar mentioning index, I finally quit fighting advice of Selvin and Ed George to use a SQLite database instead of loading all 140,000 words into memory, which I never thought was a good idea, but it worked well enough to get started. But very poorly, it turns out.

With database, the improvement is remarkable.

Indexing the only column (i.e., not overkill) made wildcard searches instantaneous by using "where word like ?" where "?" is the user's pattern.

It takes a minute to load the 140,000 words into a database (one-time task, assuming app's data isn't cleared via Settings), but it was using database.beginTransaction and endTransaction that made the loading quick enough. For details, see this.