Optimisation long collections (in LiteDB)

Question

I have a class that I store in a LiteCollection:

public class Message {
    public string Id => $"{ChannelId}-{MessageId}";
    public long ChannelId { get; set; }
    public long MessageId { get; set; }
    public string Text { get; set; }
    ...
}

MessageID can be duplicate, ChannelID is unique.

For the test, I added 700k random messages to the collection. But I think in reality there will be more.

Now it takes 70 ms to receive a specific message by identifier, and 140 ms in total from MessageID and ChannelID.

Given that the number of elements will increase, how do I optimize this?

It can not just throw everything into one collection, but, for example:

make a new collection for the channel to make a channel class, and in
it store all messages related to this channel?

There is no simple solution, you will have to manage a Dictionary for every type of lookup you want to make. One dict `ByIdentifiier` One dict with a key of `MessageId + ChannelId`. Make sure to remove items from the Lookup Dictionaries before you change MessageId / ChannelID. — Charles, Dec 31 '19 at 07:28
@Charles I understood your message, but did not quite understand how this can be implemented. Could you design as an answer and attach the code? — SKProCH, Dec 31 '19 at 07:38

Charles · Answer 1 · 2019-12-31T09:20:50.083

If all you need is a combined Dictionary Key then this will do:

var allMessages = new List<Message>()
{
    // channelId, messageId, text
    new Message(1, 1, "Message1"),
    new Message(1, 2, "Message2"),
    new Message(2, 1, "Message3")
};

var lookup = allMessages.ToDictionary(x => x.Id, x => x);

// to find an entry by (channelId, messageId)
var id = $"{1}-{2}"; // keep in sync with your Message.Id()
var myItem = lookup[id];

// make sure to remove / add an entry from the dict before / after you change it

Optimisation long collections (in LiteDB)

1 Answers1