2

I have my most of my apps "dynamic" data stored in the datastore.

However, I also have a large collection of static data that would only change with new builds of the app. A series of flat files seems like it might be simpler than managing it in the datastore.

Are there standard solutions to this? How about libraries to make loading/parsing this content quick and easy? Does it make more sense to push this data to the datastore? Which would perform better?

Anyone else have this problem and have war stories they can share?

Noel Billig
  • 91
  • 3
  • Your question here might need to be refined. What are really theses static data ? How do they really change from builds to builds ? What is the usual/natural format to store them ? – Nicolas Bousquet Mar 16 '11 at 09:32
  • Are you aware of the limit on the total size of the application and static files? See http://code.google.com/appengine/docs/java/runtime.html#Quotas_and_Limits. Did you also consider using the BlobStore to store static data? – Peter Hulsen Mar 16 '11 at 11:43
  • Hi Nicolas+Peter: The static data will be about 100-1000 rows in a spreadsheet that has about 10 columns. There will be no queries to run on this data, I would only need to access rows by id. Does pushing this into and out of the Blobstore make more sense than say, exporting each row as an xml/json file named by id? Is parsing a blob that size likely to have performance/cost implications when I may only need to access a few rows at a time? – Noel Billig Mar 16 '11 at 14:01
  • 100-1000 rows with 10 columns in a spreadsheet doesn't seem like such a large amount, or are there many such spreadsheets involved? Also, how often do you need to access the data? All the time or just once for some kind of setup? – Brummo Mar 16 '11 at 14:22
  • @Noel In that case, I would suggest storing it as a CSV in your app, and loading it into memory on startup. – Nick Johnson Mar 16 '11 at 18:00
  • Yeah, it's not a ton of data, but it still seems like a lot to parse on every request. – Noel Billig Mar 16 '11 at 22:46

1 Answers1

0

Everything depends on how you need to use the information.

I for instance have an application that needs to have a starting state provided from static data. Since I wanted this static data to be easily prepared outside the application, I put the data as spreadsheets on Google Docs and then I have an administrative function in my web app to load the starting state through Google Docs Spreadsheet API to objects in the datastore. It works fairly well, although there are some reliability issues that I haven't quite worked out yet (I sometimes need to restart the process).

In other cases, you might get away with just including the data as static property/xml files and load them through the standard Java resource APIs (getResourceAsStream and such). I haven't tried this approach though since it wasn't meaningful in my particular situation.

Brummo
  • 1,030
  • 2
  • 12
  • 18