Read huge file from disk

Question

I am developing an iOS app and I have this text file with a city name per line. I have like 3 Million cities in that file. In order to be able to perform searches and operations on it I am using a B-Tree but this tree takes a long time to be created. It is not good for the user experience having him to wait for this every time he uses the time. All this without using Core data!

Any tips on how can I speed up this process?

Thank you

score 1 · Answer 1 · answered Mar 27 '15 at 14:59

My recommendation is that you use SQLite with an index on the fields you want to query (or some other type of permanent, indexed storage) so that the user only has to wait the first time the app is opened, and then you can query the database, which will be much faster. I am also fairly certain that you can install a SQLite database from a pre-generated file, so you might be able to generate this index offline, bundle it with your application, and then the user has no wait time at all. I'm not 100% sure on this options though, so you should investigate.

Either way, there is no magic solution here. If the data you want is on line 2 million of the file, you will have to read 2 million lines of text in order to get to that line. I would recommend finding a way to make the UX of your app acceptable so that the user feels better about waiting for the data to load. If you display some sort of pretty screen with a progress bar while the data indexes, the user will be more forgiving of this wait.

if you put the text into a SQLite database you wouldn't have to search 2 million items each time you wanted the 2 millionth item. — AnthonyLambert, Mar 27 '15 at 15:10
I see that would be a good idea but unfortunately for this scenario I wont be able to use Core data or any DB. tricky! — apinho, Mar 27 '15 at 15:11
@OP, you can use SQLite without core data. There are libraries that make this easier, one I have used is FMDB (https://github.com/ccgus/fmdb), but there are many others. — aeskreis, Mar 27 '15 at 15:37

score 0 · Answer 2 · answered Mar 27 '15 at 15:00

0

The B-Tree will obviously take some time to be created. If you don't want to use a database but stick with your own B-Tree implementation you could dump the tree data to a separate file and load that when the program starts instead of recreating it every time. However, you will have to update the cached tree every time the source data is modified.

In Python the pickle module can help you, but most programming languages will have a serialisation module.

answered Mar 27 '15 at 15:00

some_developer

1
1

you can't use interpreted languages on IOS. – AnthonyLambert Mar 27 '15 at 15:02
I didn't realise it was a iOS specific question. However, serialisation is supported in Objective C according to Wikipedia: http://en.wikipedia.org/wiki/Serialization#Programming_language_support – some_developer Mar 27 '15 at 15:04

score 0 · Answer 3 · answered Mar 27 '15 at 15:08

Does this file come with the Application? If it does then you could already process the file file into an SQLite database. Before you ship the app containing the database. You can then use "Select" statements to search the data using indexed fields (like cityname).

If the file changes. Then Still ship with a database and just send amendments as a file. Which would edit the database to bring it back up to date. You may need to add a command to the file for each line like, REPLACE, NEW, DELETE.

Read huge file from disk

3 Answers3