0

I've been reading up on SimpleDB and one downfall (for me) is the 1kb max per attribute limit. I do a lot of RSS feed processing and I was hoping to store feed data in SimpleDB (articles) and from what I've read the best way to do this is to shard the article across several attributes. The typical article is < 30kb of plain text.

I'm currently storing article data in DynamoDB (gzip compressed) without any issues, but the cost is fairly high. Was hoping to migrate to SimpleDB for cheaper storage with still fast retrievals. I do archive a json copy of all rss articles on S3 as well (many years of mysql headaches make me wary of db's).

Does anyone know to shard a string into < 1kb pieces? I'm assuming an identifier would need to be appended to each chunk for order of reassembly.

Any thoughts would be much appreciated!

Joe
  • 1,762
  • 9
  • 43
  • 60

2 Answers2

0

Slicing should do it. You'd just have to keep track of the pieces.

>>> x = "big string with lots of words"
>>> x[2:10]
'g string'
>>> x[10:20]
' with lots'
>>> x[20:29]
' of words'
wolf123450
  • 103
  • 1
  • 7
0

I opted to go with storing large text documents in Amazon S3 (retrieval seems to be quick), I'll be implementing an EC2 instance for caching the documents with S3 as a failover.

Joe
  • 1,762
  • 9
  • 43
  • 60