1

I came here today with a theoretical question. (hint: it will be long and tough, but to fully understand the problem I think I have to write every important detail. If you read it to the end huge thanks for you, you're not the hero we deserved but the hero we needed)

Story time: I'm currently building an online shop from 0. It has the same principles as an ebay: users can create advertisment for their used products. The problem is that I want to create a filtering feautre.

What is my MongoDB data structure?

My page has products with different attributes, by this I mean that the products have varying categories and values. To imagine here is an example

Product A:
Creator:User1
Category:Car
Type:BMW
Color:Red

Product B:
Creator:UserB
Category:Electronics
Type:Phone
Producer:Apple

To be more complex each user can define maximum 3 more extra category and value for each product. So for example User1 adds 2 new category and the final product will be:

Product A:
Creator:User1
Category:Car
Type:BMW
Color:Red
Number of seats:4
Fuel type: Gasoline

Because of the above mentioned when a user adds a new product there will be two type of categories: the static ones which are predefined by me(Category,Type,Color -> in car's case) and the dynamic ones which the user adds (Number of seats, Fuel Type or anything else).

Overall: My final data structure in mongoDB is not static, since there are some added categories. Because of this I have a Product collection and each document looks like the above mentioned example

How are the items shown?

I have a main page. When I populate it I make a call with $skip and a $limit attribute set to 8, so for the first time I only query 8 products. If a user clicks on a Load More button it will load another 8 product and so on.

FINALLY: My actual question ...

So at this point I guess you understand everything related to the business logic so it's time for my question: if I want to filter these dynamic products, but i don't know what is the best practice for it?

My idea: First create a mongoDB collection named Categories. Each main category will be a document in it and we will store static and dynamic categories and values ex:

category:car
predefined:[{type:[BMW,Mustang,Ferrari]},{color:[red,green,blue]}]
userdefiend:[{number of seats:[2,4,5,6]},{fuel type:[Gasoline,Air,Diesel]}]

We load the the values in the main page if a user clicks a specific value ex:BMW we set a limit to 8 and go through on our Product collection and get the 8 items which has a Type:BMW. If he selects another option ex: color:Red we loop again through the collection but now with two criteria: Type:BMW and color:Red.

Idea2: Create a Category collection again with this structure

categoryType:predefined
mainCategory:Car
categoryName:Type
BMW:[prodA, prodC,prodD]
Ferrari:[prodD,prodE]
...values:products which contains

categoryType:userdefined
mainCategory:Car
categoryName:Number of seats
4:[prodA, prodD],
5:[prodE]

If a user selects from Type category the BMW we load the products from the BMW fields [prodA,prodC,prodD]. If the user selects Number of seat category with a value 4 we load the [prodA, prodD] and on the webpage we use a filter with our actual products so it remains only [prodA,prodD]. And from our actual list we use findById for the specific products.

I think that these are not the best options from any perspective, but I am really confused.

What do you guys think how should I structure my categories/products to have an efficent read/write/update complexity?

Anyways thank you for reading this and if you made it until here I'm curious about your idea. Have a nice day

UPDATE:

The filtering functionality

To don't have any confusion this is my filtering idea: When a user selects a main category for example Car or Electronics I want to show only the relevant filtering categories and options. Filtering categories in Car's case are Type and Color. I want these filtering options to have pre-poupulated options. By this I mean, that if a filtering category is Type, and there are 2 Products which has Type:BMW and Type:Ferrari I want to show these values as options for filtering. And I don't want to hardcode these options, for example I hardcoded Type:Laborghini and I have no products with type Laborghini.

By the end if a user clicks to a Type:BMW I will filter all of my products based on that criteria.

My filtering side menu will look like this:

Type: BMW,Ferrari (these values exists in my database)
Color:Red,Black,Grey,Yellow

And for user-added categories I will build a searchbar, if a user selects a userdefiened category I want to add to the filtering categories so the overall look would look like this:

Type: BMW,Ferrari (these values exists in my database)
Color:Red,Black,Grey,Yellow
Number of seats:4,6,7 (number of seats category is added by user, 4,6,7 are the existing values to this category)
MrNobody
  • 535
  • 8
  • 24
  • I would build upon the dynamic nature of Mongo. All items in a collection, no categories collection. Throwing out performance on the window. https://stackoverflow.com/a/43570730/1859959 <- this way you can check on the actual "columns" users ended up with. Whenever a filter is added, you can rerun this aggregate to filter the "columns" more. There would be a template collection for your predefined categories, but instantiating a row, the user adds categories what he/she wants. – ntohl Apr 23 '21 at 10:42
  • First of all thank you for reading my question.I appreciate your answer,but the last part about:"There would be a template collection for your predefined categories,but instantiating a row,the user adds categories what he wants" it's not clear what do you mean by that. I understand that if a user selects a main category:Car we use an aggregate and get all the "columns" (filtering category) which matches the Car.But if we want to show for a column the relevant values(aka filetring options="rows")without user instantiating how can we do it?Maybe you can add an answer,because it's a good idea. – MrNobody Apr 23 '21 at 12:09

1 Answers1

1

You could structure Your data like having a generic Products collection. Having both

Product A:
Creator:User1
Category:Car
Type:BMW
Color:Red

Product B:
Creator:UserB
Category:Electronics
Type:Phone
Producer:Apple

rows. Whenever you show the filter component, you can select the available categories by using an Aggregate (https://stackoverflow.com/a/43570730/1859959)

This would generate search boxes like "Creator", "Category", "Type", "Color", "Producer".

The data itself would be as generic as possible.

When the user wants to add a new product, it starts out from a template, like "Car" or "Electronics". The Templates collection gives him the initial values, which should be included. So it would be like:

{Car: [{type:[BMW,Mustang,Ferrari]},{color:[red,green,blue]}],
Electronics: ... }

Selecting a Car would generate the "type" and "color" input boxes. Saving the form would insert the new row into Products.

ntohl
  • 2,067
  • 1
  • 28
  • 32
  • Ah, okay I got it now, thank you :) I have already done the same way you mentioned the templating and the Product collection too. My question is, that if I have for example 200k products and I do the aggregate on my Product collection to show the filtering categories and options won't be too slow if I go through all of the products? – MrNobody Apr 23 '21 at 12:50
  • You could Memoize/Cache the results for the Aggregate query. That would speed things up. But you have to measure first. – ntohl Apr 23 '21 at 13:03
  • I have implemented it, but it's so slow for big datas. Do you have any other idea? – MrNobody Apr 28 '21 at 12:06
  • Sorry to hear that. Maybe https://github.com/sebmaldo/functional-memoize#readme this could help, but I don't know how much memory would it consume... https://docs.mongodb.com/manual/faq/fundamentals/#does-mongodb-handle-caching- sais "MongoDB does not cache the query results in order to return the cached results for identical queries" which I understand as it's not caching the query plan. Indexes set? – ntohl Apr 28 '21 at 12:16
  • Yup, for the static fields I set it.This is the aggreage function which I tried: https://stackoverflow.com/questions/67237111/mongodb-group-all-keys-and-values-in-a-collection-by-a-certain-field . I guess I will try with references – MrNobody Apr 28 '21 at 12:19
  • Sorry I don't know how would an aggregate behave in case of caching/index usage. If references work, I'm happy for you. I think the functional-memoize link is client side. It might consume too much memory, and have cache-invalidation problems. – ntohl Apr 28 '21 at 12:39
  • @MrNobody I would do that sort of filtering server side where you have extra cores to work with and can multithread until your heart's content. You tagged this post with reactJS which means client side you're using JavaScript. JS doesn't natively multithread hence why async is so important on the client. Server side if you had a Java app you could just do something like `products.parallelStream().filter(product -> product.category.equals("mycat")).collect(Collectors.toList());` – TheFunk Apr 29 '21 at 19:24
  • @TheFunk thank you for the advice I will check further about this idea. – MrNobody Apr 29 '21 at 20:30