Dictionary: hard-coded vs. external file

Question

I have a java application which is started and stopped multiple times per second over hundreds of millions of items (called from an external script).

Input: String key
Output: int value

The purpose of this application is to look for a certain key in a never ever ever changing Map (~30k keys) and to return the value. Very easy.

Question: what is more efficient when used multiple times per second:

hard-coded dictionary in a Map

Read an external file with a BufferedReader

...amaze me with your other ideas

I know hard-coding is evil but sometimes, you need to be evil to be efficient :-)

It would be better to have the map on the fly for easy access regarding your Java app. — Luiggi Mendoza, May 02 '13 at 19:55
Why are you starting and stopping the app so much? Why not just leave it running and process requests as needed? — dlev, May 02 '13 at 19:58
The problem here is the fact that you relaunch your Java App hundreds of millions of times. The startup overhead is likely to be 99% of the processing time. You need to rethink things so your app can service requests over a pipe or socket and startup only once. — Jim Garrison, May 02 '13 at 19:59

score 2 · Accepted Answer · answered May 02 '13 at 19:58

2

Read in the dictionary from file. Store it in a Map. Set up your Java application as a service that runs continuously (since you said it gets called many times per second). Then your Map will be cached in RAM.

answered May 02 '13 at 19:58

KyleM

4,445
9
46
78

score 1 · Answer 2 · answered May 02 '13 at 21:01

The fastest is a hard coded map in memory. if u a have a huge file you can use a Memory Mapped file :

MappedByteBuffer in = new FileInputStream("map.txt").getChannel().map(
FileChannel.MapMode.READ_ONLY, 0, LENGTH);
StringBuilder bs = new StringBuilder();
//read 1/4 of the file   
while (i < LENGTH/4)
      bs.append((char)in.get(i++));

This approach is a bit problematic though,in practice you will want to partition the file on line breaks i.e read until the 100th line clean the buffer and read some more.

score 0 · Answer 3 · answered May 02 '13 at 19:57

I would load the file into a Map at startup of the application, and then use it as you describe.

I would store the data in a database for faster load times.

Definitely do not have the application startup and shutdown every time it is called; have it as a service that waits for IO, using Asynchronous I/O, such as netty

Dictionary: hard-coded vs. external file

3 Answers3