How to store millions of children in 100 parents in mongoDB

Question

I am developing a Node.js server that will add children to 100 different parents every 60 seconds.

The parents are 100 different markets:

and the children are timestamps containing some data in them:

      ...millions of more timestamps

Timestamps will be more than millions in the database. I am thinking of Array on Ancestors to store an array of children(timestamps) in every parent(only 100), so I'm thinking the arrays will contain millions of objects. Is that safe?

I am using mLab and followed the tutorial on MongoDB Docs to create the database using:

db.collection('database').insert({ _id: "data", ancestors: [], parent: null })
db.collection('database').insert({ _id: "ADABTC", ancestors: ["data"], parent: "data" })
db.collection('database').insert({ _id: "entry", ancestors: ["ADABTC", "data"], parent: "ADABTC" })

but I ended up with something that I could not understand. I am very familiar with Firebase but got confused trying to switch from Firebase to MongoDB because of costs when storing mass data.

Which methods can I use in Node.js to achieve this data structure?

That's not the only suggested structure for "trees". It's a [whole chapter of different approaches](https://docs.mongodb.com/manual/applications/data-models-tree-structures/), and some scale to larger amounts of nodes than others. It's honestly not up to anyone else but you to determine which one best suits your application use case. So there is no *"one best option"*. — Neil Lunn, Apr 16 '18 at 04:26

score 0 · Answer 1 · answered Apr 17 '18 at 04:06

As the comment says, there is no one right answer here. How you are going to add records to, and retrieve data from the mongo datastore will influence the decisions. I couldn't give this suggestion in the comments because I wanted to show a formatted example. So posting as an answer, even though I don't mean it to be the one right answer.

You might want to consider setting this up flat, like the below. Speed then becomes a matter of indexing. They are easy and cheap to create. The below would probably want an index on name and one on name, timestamp. Here are details on how to do that: https://docs.mongodb.com/manual/reference/method/db.collection.createIndex/

{
  {
    name: "ADABTC",
    timestamp: 1521421385793,
    close: 0.00001962,
    high: 0.00001962,
    etc...
  },
  {
    name: "ADABTC",
    timestamp: 1521421385793,
    close: 0.00001962,
    high: 0.00001962,
    etc...
  },
  {
    name: "AIONBTC",
    timestamps: 1521421385793,
    close: 0.00001962,
    high: 0.00001962,
    etc...
  },
  etc...
}

If this becomes too large (Mongo can easily handle millions of records), then you might want to split each market into it's own collection. Though that could be less ideal, based on your query / reporting criteria.

How to store millions of children in 100 parents in mongoDB

1 Answers1