0

I've created an API for some entities, each of which have their own schema and a corresponding collection. Such as:

posts
authors
comments

There is one to one relationship between posts and authors, and one to many relationship between posts and comments.

Now the data size is increasing, and we are planning to distribute the data customer wise to speed up query and aggregations. This will mean we will now have following collections:

<customer_1>_posts
<customer_2>_posts
...
<customer_n>_posts

<customer_1>_authors
<customer_2>_authors
...
<customer_n>_authors

<customer_1>_comments
<customer_2>_comments
...
<customer_n>_comments

Since structure is common across all <customer_x>_posts, and similarly across corresponding authors and comments, I'm thinking of using common schema and API.

Now, is there a way to use common Mongoose models and decide the collection for data storage at the save time, depending on the value of the customer field? This would save me from creating a separate model for each client, and also from creating separate endpoints.

Gaurav
  • 347
  • 4
  • 12

1 Answers1

1

You can dynamically build your models, and also dynamically look up and use your models.

Where you create your models:

for (const customer of customerList) {
  mongoose.model<CompanyDocument>(`${customer}_comments`, schema)
}

Then to utilize:

CustomerModel = mongoose.model(`${customer}_comments`)

That said - while this is definitely doable - this is not a good pattern. Mongo should scale for all of your use cases if you properly use indices, with no need for this added complexity. Millions of posts and comments can still be fast and sustainable without breaking into separate collections.

AdamExchange
  • 1,253
  • 1
  • 8
  • 16
  • Thanks @AdamExchange But this pattern would still mean modifying the customers list every time a new customer is added, and also adding repetitive code to handle that particular customer's posts etc. Data separation for multi-tenancy is another reason for which separate collections are sought. – Gaurav Oct 28 '22 at 11:48
  • The customer's list can be saved in Mongo as well, so you wouldn't need any code modification for this pattern. I've used a similar pattern, but we were separating clients in separate databases instead of separate collections for data isolation. – AdamExchange Oct 29 '22 at 02:45
  • That's what I'm supposed to do as well. I've suggested client specific collections as an alternative. How did you manage connections in that case, specially when new clients are added? – Gaurav Nov 01 '22 at 03:33
  • There was a lookup table in an admin database that held a clients subdomain and database name. When new clients signed up, they were automatically added to that lookup table. Then when an incoming request was made, we had middleware that used that lookup table (it was cached, so it only hit the database if it was a client it hadn't seen before). I am not able to post the whole code on how then it connected to the right database, but we used some of the tricks that were laid out here: https://stackoverflow.com/questions/22588253/dynamic-database-connection-to-mongodb-or-mongoose-from-nodejs – AdamExchange Nov 01 '22 at 17:05