1

I have a Sinatra application that needs some static data (~15'000 key/value-pairs). The data does only change every few months or so. I do not want to store the data in a database (the app does not use any other data storage and I dont't want to add any just because of this).

I currently have the following solution:

data/invTypes.yml:

18: "Plagioclase"
19: "Spodumain"
20: "Kernite"
21: "Hedbergite"
22: "Arkonor"
25: "Corpse"
34: "Tritanium"
35: "Pyerite"
36: "Mexallon"
37: "Isogen"

And then in my app.rb I do something like this:

# load Type IDS from Static Data
IDS = YAML.load(open('data/invTypes.yml').read

Is there a better solution?

mhutter
  • 2,800
  • 22
  • 30
  • "is there a better solution" - What don't you like about this one? – Sergio Tulentsev Apr 17 '15 at 12:02
  • I worry about my memory footprint once the dataset gets bigger or more data is added. – mhutter Apr 17 '15 at 12:04
  • The way to combat memory bloat is to not load stuff you don't need (now). How is this applicable to your app - I have no way to know (except you telling me all about it) – Sergio Tulentsev Apr 17 '15 at 12:07
  • For example: If some items are used much more frequently than others, put them in a separate file. So that you won't need to load all the "cold" keys when you need only one of the "hot" ones. – Sergio Tulentsev Apr 17 '15 at 12:08
  • 2
    15000 k/v pairs? This is, like, 100kb of data. If your machine is so underpowered that you have to count kilobytes, you have chosen a wrong technology for the app, my friend. :) – Sergio Tulentsev Apr 17 '15 at 12:11
  • It does'nt seem like much but the dataset ist potentionally much much larger ;-) – mhutter Apr 17 '15 at 12:28
  • 1
    *Potentially* you'll be using a database :) – Sergio Tulentsev Apr 17 '15 at 12:31
  • 2
    On a larger dataset you’ll be interested in quick data access based on queries. SQL is proven to be a winner in this area. That said, the very first problem you’ll meet on potentially huge dataset is _not_ a memory consumption, is a _data access time_. – Aleksei Matiushkin Apr 17 '15 at 12:46
  • You can try doing a JSON file. Regardless, this should work properly as well. I guess it depends on how much overhead you get from reading the file. You could even do a CSV file and just split on commas. There are many ways to do this, it just depends on your needs. YAML should work just fine though. Also, seems that you are playing eve-online? – Phobos Apr 21 '15 at 23:57
  • There is nothing wrong with putting this data in RAM (since it's fast, and you have enough of it), the only thing I would optimize is to make sure that you parse the file only if necessary and not on every request. Maybe something like this: http://stackoverflow.com/questions/4525482/in-sinatraruby-how-should-i-create-global-variables-which-are-assigned-values – Sir l33tname Apr 24 '15 at 07:28

0 Answers0