Storing Data - Big list into entity properties

Question

I run a crawler back-end in my application that everyday mines some websites for data.

For every website I crawl I create an entity, stores a big list of String id's.

The way I do it right now is with a repeated ndb.StringProperty() not indexed.

After 3 days of run it consumed 70% of my datastore storage.

What could be the next thing to do? Store them as compressed json?

Store them in the blobstore and everytime read the blob, etc?

Something else? An alternative?

@PaulC yes I am trying that with Json at the moment because I operate on lists. — Jimmy Kane, Jan 23 '13 at 15:29

score 0 · Accepted Answer · answered May 13 '13 at 16:01

0

Compressed and JSON did the trick for me. Closing this.

answered May 13 '13 at 16:01

Jimmy Kane

1 Answers1