1

I wonder if it's a good/bad idea to use deepstream record.getList for storing a lot of unique values, for example, emails or any other unique identifiers. The main purpose is to be able to answer a question quickly whether we already have, say, a user with such email (email in use) or another record by specific unique field.

I made few experiments today and got two problems: 1) when I tried to populate the list with few thousands values I got 

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

and my deepstream server went off. I was able to fix it by adding more memory to the server node process with this flag

--max-old-space-size=5120

it doesn't look fine but allowed me to make a list with more than 5000 items.

2) It wasn't enough for my tests so I precreated the list with 50000 items and put the data directly to rethinkdb table and got another issue on getting the list or modifing it:

RangeError: Maximum call stack size exceeded

I was able to fix it with another flag:

--stack-size=20000

It helps but I believe it's only matter of time when one of those errors appear in production when the list size reaches proper value. I don't know really whether it's nodejs, javascript, deepstream or rethinkdb issue. That's all in general made me think that I try to use deepstream List wrong way. Please, let me know. Thank you in advance!

Vasiliy Sadokhin
  • 87
  • 6

1 Answers1

2

Whilst you can use lists to store arrays of strings, they are actually intended as collections of recordnames - the actual data would be stored in the record itself, the list would only manage the order of the records.

Having said that, there are two open Github issues to improve performance for very long lists by sending more efficient deltas and by introducing a pagination option

Interesting results in regards to memory though, definitely something that needs to be handled more gracefully. In the meantime you could drastically improve performance by combining updates into one: 

var myList = ds.record.getList( 'super-long-list' );

// Sends 10.000 messages
for( var i = 0; i < 10000; i++ ) {
    myList.addEntry( 'something-' + i );
}

// Sends 1 message
var entries = [];
for( var i = 0; i < 10000; i++ ) {
    entries.push( 'something-' + i );
}

myList.setEntries( entries );
wolframhempel
  • 1,094
  • 10
  • 12
  • My case is that I can't do bulk update as, for example, user will sign up one by one, so I will get new email in list per registration. Let's stick to sign up with unique email example as it's obvious but note that we have other entries with unique values I need to care about so we won't have duplicates. I wonder if list is not a right architectural decision to handle this business logic. Two other solutions come in mind: deepstream.io-provider-search-rethinkdb and look at rethinkdb directly to search by filter or secondary index. Any other ideas? – Vasiliy Sadokhin Mar 20 '16 at 01:32
  • 1
    Usernames and emails are immutable - they don't really change. I think it would make sense for them to be stored in the DB using a traditional request/response based approach via RPCs. The challenge with deepstream.io (and any other realtime framework for that matter) is to be efficient in the use of realtime data. That means identifying data that's a) displayed in the current view b) subject to (reasonably frequent) change which for usernames might not necessarily the case. That's why deepstream provides its mixture of RPC, events and records – wolframhempel Mar 21 '16 at 20:27
  • Yeah, thank you for you reply. I do agree that username and other fields I'm going to have unique won't be changed often so I stick to direct lookup in database. – Vasiliy Sadokhin Mar 21 '16 at 22:36