3

we have started using elasticsearch in our project, we are storing user data and his friend list as nested object, and nested to nested object storing friend's friend list because we required this data when we are doing global search. Now we are syncing this data in real time with our database, so is this good to syncing done in real time 50-100 TPS or in future it will create problem.

We need to create complex queries for updating the data because we are managing friend list in 2nd level. so how to create advance scripting in painless, I have checked this in Google but not found anything in detail.

If my approach is wrong of doing this, please let me know.

Deepesh Uniyal
  • 923
  • 3
  • 20
  • 44
  • 1
    Your questions are vague and open ended (`how to create advance scripting in painless`). Be more specific! Regarding your nested data structure with friends-of-friends-of-friends-of..... will get you trouble some day. – Andrei Stefan Nov 09 '17 at 13:45
  • Hi @AndreiStefan, Please suggest me what is the right way of storing this data. – Deepesh Uniyal Nov 10 '17 at 04:13
  • 1
    Not in elasticsearch, is it? Did you read about graph databases? They may suit your use case. – Archit Saxena Nov 14 '17 at 06:35
  • 3
    Have you tried anything yet? Could you edit your question and provide a snippet of your code to show what you have tried, and why it didn't work. Also including some schema configuration information, or a graphic showing how your items are related would help. – Phil Nov 15 '17 at 11:41
  • I was in wrong direction, now I know that we have to use some database for same, Neo4j or Orient DB. Can you please suggest which database will fit for social media project where people are connected with each other. One more question I need the n-gram(for auto-complete search) facility so choosing the database I have to consider the same also. – Deepesh Uniyal Nov 16 '17 at 02:30
  • @DeepeshUniyal I don't think a NOSQL db is what you are looking for. You can look for a graph db or a traditional relational db. Sorry to say your question denotes no intimace with the technologies you are using (not uncommon) but it means you will need to take a time to study, learn and make some "lab work" (build small experimental projects) – jean Nov 16 '17 at 10:33
  • @jean My main requirement is n-grams searching, which giving by elasticsearch or solar, How can I achieve this with traditional database. – Deepesh Uniyal Nov 16 '17 at 10:55
  • @DeepeshUniyal The friend list and friendship relations can be easily done with in any relational db. Since you are searching for the 2nd level you will not fall in the only thing relational DB are really poor: recursion. You still can use a NOSQL db to solve some kind of searchs but for the friendship you can really give a look in graph db and mayyybe relational db – jean Nov 16 '17 at 11:01

1 Answers1

2

To answer your first question

so is this good to syncing done in real time 50-100 TPS or in future it will create problem

As of version ES6.0, Multi level nesting is automatically supported, and detected, resulting in an inner nested query to automatically match the relevant nesting level (and not root) if it exists within another nested query. But there is a caveat, indexing a document with 100 nested fields actually indexes 101 documents as each nested document is indexed as a separate document. To safeguard against ill-defined mappings the number of nested fields that can be defined per index is usually limited to 50 using the index.mapping.nested_fields.limit . This setting allows you to limit the number of field mappings that can be created manually or dynamically, in order to prevent bad documents from causing a mapping explosion. So to answer your question, this is fine, but as your data grows, it becomes more complicated to manage and you risk the danger of a mapping explosion.

To answer your second question

We need to create complex queries for updating the data because we are managing friend list in 2nd level. so how to create advance scripting in painless, I have checked this in Google but not found anything in detail.

You might need to present some context here to be able to understand why your approach is necessary, but basically, in a social profile context, managing a friends list as you are doing is always a bad idea, especially if you anticipate scaling in the future. It may work for smaller use-cases, but it does not work very well when you scale. This is because, the relationships become more sophisticated and you will end up having too many multi nested objects. As mentioned, all factors kept at a constant, you might want to look at a graph database for this kind of a scenario. You could, however, have other reasons to your approach which is why you might want to enumerate your context so we can better advise.

Hope this helps!!

kioi
  • 339
  • 2
  • 15
  • Hi @Kioi Thanks for answering. I have checked the graph database and found the Neo4j and OrientDB but another problem is n-grams searching(autocomplete). In elastic search or solar we can achieve this, but if I will fetch the data from graph database its not possible. – Deepesh Uniyal Nov 17 '17 at 02:33
  • Hope it presents a satisfactory response. – kioi Nov 17 '17 at 02:34
  • Hi @Kioi Thanks for answering. I have checked the graph database and found the Neo4j and OrientDB but another problem is n-grams searching(autocomplete). In elastic search or solar we can achieve this, but if I will fetch the data from graph database its not possible. – Deepesh Uniyal Nov 17 '17 at 05:07
  • @DeepeshUniyal technically, n-gram searching as I know it, would be very difficult to do on a graph database, largely because of how graph databases are structured. NEO4j for example, has an autocomplete feature, but this is not as effective as on elastic and solar. For NEO4j for example, there are a couple of workarounds around this problem (most involving algorithms working outside the db and then sending the results to the db), details of which I may not be able to explain here, but they all depend on how your project is structured and what you are trying to achieve. – kioi Nov 17 '17 at 13:57
  • I can sort of see what you are trying to achieve here, but I would suggest you rethink how you are indexing your data and where, to be able to achieve both end results or sacrifice one for the other. Also, the kind of infrastructure you are running really matters, to be able to determine how you will structure your solution. – kioi Nov 17 '17 at 14:00
  • Hi @kioi, I am glad that you know what I want to achieve, Now I am using mysql database and the data which we need to N-Gram searching we are syncing with elasticsearch at real time. – Deepesh Uniyal Nov 19 '17 at 06:08
  • That is an interesting architecture, reach out to me and we can discuss this architecture further and what your end goal is. Regards – kioi Nov 22 '17 at 06:47