How to insert millions of keys/values in Redis using Multithreaded C# application

Question

I need to create a C# application (Windows Service), which runs every 5 seconds (interval), generates around 20 Million values.

I need to insert these 20 Million values into Redis (1 Key / Value) in under 5 seconds, making sure the inserts are finished before the next interval is started.

Note: I have to only keep 7 cycles in Redis => 20 Million * 7 => 140 Million Keys in Redis

I am using C#'s Threading.Tasks to call a function (20 Million Times), so that they are processed in parallel (asynchronously).

I have even created a pool for Redis clients for my process to be able to execute Redis queries also in parallel.

Here is the C# part calling the function 20 Million times:

List<Task> tasksList = new List<Task>();

foreach (object k in ListOf20MillionData)
{
    tasksList.Add(

        Task.Factory.StartNew(() =>
        {
            GenerateValue(k);
            //Inside 'GenerateValue' data is generated and pushed to redis
        })
    );
}

Here is a section of code inside 'GenerateValue' which gets a redis client object from a Pool of clients, executes the insert and releases back the redis client into the pool.

RedisClient redisClientObj = RedisPool.GetNextAvailableClient();

redisClientObj.Add("SomeKey", "SomeValue");

RedisPool.ReleaseRedisClient(redisClientObj );

My Concerns and challenges:

1. Is my concept of Redis Pools ok?
2. How many Client connections can Redis handle?
3. Is my request even possible to achieve using C# and Redis?
4. Any advice or recommendation is highly appreciated.

Answer 1

1. Is my concept of Redis Pools ok?

Not really. Pools don't give you more throughput. They break up different logical connection scopes re sequential commands, and they allow simple concurrency... but the redis core is single-threaded, and you should be looking to saturate network, not threads.

2. How many Client connections can Redis handle?

Tons, but adding more won't help you if you can't saturate them - in fact, having lots of connections increases the overheads.

3. Is my request even possible to achieve using C# and Redis?

Only on very beefy boxes with a huge network; you might increase throughput with "cluster", but that also increases packet fragmentation

4. Any advice or recommendation is highly appreciated.

Batch. Batch like crazy, to minimize round-trips. Fat batches with a tiny responses makes very effective use of the network, and don't require you to have complex code. And the redis mset command is optimized for exactly that: fat batches with tiny responses.

Locally, with the same machine inventing the data on a single thread and being the redis server, for me it still takes 34 seconds, though:

    static void Main()
    {
        using (var conn = ConnectionMultiplexer.Connect("127.0.0.1:6379"))
        {
            var db = conn.GetDatabase();
            var watch = Stopwatch.StartNew();
            foreach(var batch in InventData(20000000).Batchify(5000))
            {
                db.StringSet(batch);
            }
            watch.Stop();
            Console.WriteLine(watch.ElapsedMilliseconds);
        }
    }

or if I use Parallel, i.e.

            var watch = Stopwatch.StartNew();
            Parallel.ForEach(InventData(20000000).Batchify(5000),
                batch => db.StringSet(batch));
            watch.Stop();

it takes 16 seconds.

and (see comments) if I combine Parallel with async:

            var watch = Stopwatch.StartNew();
            Parallel.ForEach(InventData(20000000).Batchify(5000),
                batch => db.StringSetAsync(batch));
            watch.Stop();

then it takes just under 14s.

with

    static IEnumerable<KeyValuePair<RedisKey, RedisValue>> InventData(int count)
    {
        if (count < 0) throw new ArgumentOutOfRangeException(nameof(count));
        string dictionary = "abcdefghijklmnopqrstuvwxyz _@:0123456789";
        int dLen = dictionary.Length;
        var rand = new Random(12345);
        const int KEY_LEN = 10, MAX_VAL_LEN = 50;
        char[] keyData = new char[KEY_LEN];
        char[] valueData = new char[MAX_VAL_LEN];
        while (count-- != 0)
        {
            for (int i = 0; i < keyData.Length; i++)
                keyData[i] = dictionary[rand.Next(dLen)];
            var len = rand.Next(10, MAX_VAL_LEN);
            for(int i = 0; i < len; i++)
                valueData[i] = dictionary[rand.Next(dLen)];

            yield return new KeyValuePair<RedisKey, SomeType>(
                new string(keyData), new string(valueData, 0, len));
        }
    }

    static IEnumerable<T[]> Batchify<T>(this IEnumerable<T> source, int batchSize)
    {
        var batch = new List<T>(batchSize);
        foreach(var item in source)
        {
            batch.Add(item);
            if (batch.Count == batchSize)
            {
                var arr = batch.ToArray();
                batch.Clear();
                yield return arr;
            }
        }
        if (batch.Count != 0) yield return batch.ToArray(); // trailers
    }