2

I need a way to exceed the NoSQL document item size limitations while preserving read/write performance.

All NoSQL implementations have document item size limits (e.g. AWS DynamoDB is 400KB, MongoDB is 16MB). Moreover, the larger the document item size, the slower the retrieval of the document becomes. Read, write, update and delete performance is important in my implementation.

I'm already using a NoSQL database and the documents I work with have a tree structure with four levels (The document describes vector drawings of varying sizes and complexity). Each node of the tree is a component that has an ID and a body that refers to the below components. Reading and updates can be performed for the full tree or parts of the tree.

I've considered switching to a Graph database but they are optimized to querying relationships in graphs. What I need a simple and high performance way to store an expanding tree documents rather than handling graphs.

I'm thinking that I need an implementation of Closure Table structure in NoSQL to split the tree nodes into separate components and - thus - breaking the document size barrier. On the client side, a ORM (object-relational mapper) with lazy-loading will retrieve the document tree nodes as necessary.

My questions are:

  • Has this been implemented before?
  • Is there another way to get the functionality described above?

If you need such a functionality, upvote this so that more exports would answer this question.

Fahad Alrashed
  • 1,272
  • 2
  • 11
  • 17

1 Answers1

1

DynamoDB does not have penalty of speed for size, yet networking will take extra time, that's it. Also if you have such a big file and trying to push it to DynamoDB, that's the design issue to begin with. Just store big file in S3 and store meta data in DynamoDB.

You are paying for write & read units by KBs with DynamoDB, why waste money on big data when S3 is so cheap for it and has the size limit is 5TB instead of 400KB. Also if your app generates those big JSON structures, you could save $ overtime by using S3 storage classes automation, which would move unused JSON to "colder" storage class.

Lukas Liesis
  • 24,652
  • 10
  • 111
  • 109
  • Storing JSON on S3 doesn't help read, write, update and delete performance of structures within the JSON objects. As I said in the question, I have a JSON with a tree structure and nodes in the tree may be updated alone or in batch. Closure tables helps with that part. My question is: Is there a closure table implementation in NoSQL? or something else? – Fahad Alrashed Jan 23 '21 at 11:58
  • 1
    i think you should check this video: https://www.youtube.com/watch?v=HaEPXoXVf2k sounds like you are doing something fundamentally wrong at data access design level which is a bit hard to understand w/o full context. Just check that video, it should provide new thoughts for you and look fresh at your data structure. – Lukas Liesis Jan 23 '21 at 18:41